How to operator Kubernetes cluster in LocalStack EKS?

Platform
1

When I create an EKS cluster with LocalStack:

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"

Confirm:

awslocal eks describe-cluster --name cluster1
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-22T13:52:00.327000+09:00",
        "version": "1.22",
        "endpoint": "https://localhost.localstack.cloud:4511",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "ACTIVE",
....
}

Set kubernetes context:

awslocal eks update-kubeconfig --name cluster1 && \
    kubectl config use-context arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1

Why got error when run kubectl command?

kubectl get po
I0522 14:04:44.688006   27587 versioner.go:58] Get "https://localhost.localstack.cloud:4511/version?timeout=5s": EOF
E0522 14:04:55.119358   27587 memcache.go:265] couldn't get current server API group list: Get "https://localhost.localstack.cloud:4511/api?timeout=32s": EOF

The EKS’ endpoint is using 4511. The LocalStack’s default endpoint is 4566.
Is the port a reason?

2

Hi @seikyo-cho-lvgs,

For specific services, ports are assigned from the 4510-4559 range. You can see which ports are available in the docker-compose.yml.

Please check your local configuration of the .kube\config file. For a quick check, create a backup and then remove it and try again, creating the EKS in LocalStack.

3

Hi @Marcel ,

Thank you for your reply.
I didn’t use docker-compose but created the EKS cluster by this guide:

About the port in my .kube/config, it’s 4511.

apiVersion: v1
kind: Config
clusters:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    cluster:
      server: https://localhost.localstack.cloud:4511
      certificate-authority-data: very-long-string
      insecure-skip-tls-verify: false
users:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        args:
          - --region
          - ap-northeast-1
          - eks
          - get-token
          - --cluster-name
          - cluster1
          - --output
          - json
        command: aws
contexts:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    context:
      cluster: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
      name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
      user: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
preferences: {}
current-context: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1

The port is in the range, but why can’t conenct?

4

I am unable to reproduce your issue with the shared details.

The EKS and the .kube/config file is correctly created,

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: very-long-string
    server: https://localhost.localstack.cloud:4510
  name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
contexts:
- context:
    cluster: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    user: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
current-context: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      command: aws

I am able to invoke the kubectl afterwards

kubectl cluster-info
Kubernetes control plane is running at https://localhost.localstack.cloud:4510
CoreDNS is running at https://localhost.localstack.cloud:4510/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://localhost.localstack.cloud:4510/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

Can you please confirm in what type of environment are you running LocalStack?
Are you changing the default configuration, as shown in docker-compose.yml?

5

Hi @Marcel ,

I’m using Kubernetes in Rancher Desktop. I started LocalStack by CLI:

localstack start

Maybe clear the .kube/config can fix the issue:

bug: Not able to connect to an eks cluster on localstack via kubectl · Issue #10456 · localstack/localstack · GitHub

But I can’t confirm it now since when I create a new cluster this time, got status FAILED:

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-24T16:04:50.704000+09:00",
        "version": "1.22",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "CREATING",
        "platformVersion": "eks.5",
        "tags": {}
    }
}

awslocal eks describe-cluster --name cluster1
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-24T16:04:50.704000+09:00",
        "version": "1.22",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "FAILED",
        "platformVersion": "eks.5",
        "tags": {}
    }
}

I don’t know why.

6

After I delete arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1 context in .kube/config, I can create the EKS cluster again.

localstack start

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"

But when I run the kubectl command:

kubectl get nodes

I got this error:

E0527 11:08:27.358638   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:27.642766   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:27.932002   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:28.215254   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:28.509132   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

When I run the command with -v

kubectl get nodes -v

The details is

kubectl get nodes -v=10
I0527 11:07:01.291100   30213 loader.go:395] Config loaded from file:  /Users/test/.kube/config
I0527 11:07:01.291405   30213 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:01.589791   30213 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost.localstack.cloud resolved to [{127.0.0.1 }]
I0527 11:07:01.590093   30213 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:4510 succeed
I0527 11:07:01.888528   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 597 milliseconds
I0527 11:07:01.888560   30213 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 3 ms ServerProcessing 5 ms Duration 597 ms
I0527 11:07:01.888564   30213 round_trippers.go:577] Response Headers:
I0527 11:07:01.888569   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:01.888574   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:01.888576   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:00 GMT
I0527 11:07:01.888578   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:01.888580   30213 round_trippers.go:580]     Audit-Id: d7d94a11-dab3-438c-87e3-a45ebff604a7
I0527 11:07:01.889035   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:01.889291   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:01.889304   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:01.889391   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.179045   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 289 milliseconds
I0527 11:07:02.179075   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 289 ms
I0527 11:07:02.179079   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.179085   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.179087   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.179089   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:00 GMT
I0527 11:07:02.179091   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.179093   30213 round_trippers.go:580]     Audit-Id: 4db51446-b5be-4109-977e-140d55557414
I0527 11:07:02.180338   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.180593   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.180604   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.180609   30213 shortcut.go:103] Error loading discovery information: the server has asked for the client to provide credentials
I0527 11:07:02.180675   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.470294   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 289 milliseconds
I0527 11:07:02.470318   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 289 ms
I0527 11:07:02.470324   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.470328   30213 round_trippers.go:580]     Audit-Id: 7733bff9-2e86-4a35-8b40-82adfa2c0531
I0527 11:07:02.470331   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.470334   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.470336   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:02.470340   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.470807   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.471059   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.471072   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.471153   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.756941   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 285 milliseconds
I0527 11:07:02.756975   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 285 ms
I0527 11:07:02.756978   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.756984   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.756986   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:02.756989   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.756991   30213 round_trippers.go:580]     Audit-Id: a6462cbd-e99d-4214-9111-157462643de9
I0527 11:07:02.756994   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.757417   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.757670   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.757680   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.757746   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:03.053843   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 296 milliseconds
I0527 11:07:03.053871   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 296 ms
I0527 11:07:03.053875   30213 round_trippers.go:577] Response Headers:
I0527 11:07:03.053880   30213 round_trippers.go:580]     Audit-Id: ec721429-86b8-47ad-acc0-79f39a3431cc
I0527 11:07:03.053897   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:03.053903   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:03.053905   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:03.053906   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:03.054261   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:03.054550   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:03.054565   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:03.054716   30213 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials",
  "reason": "Unauthorized",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Why the server asked for the client to provide credentials?
Doesn’t it provide by default? How to provide?

7

Hi @seikyo-cho-lvgs,

The new error corresponds with the one from the GitHub issue bug: Not able to connect to an eks cluster on localstack via kubectl · Issue #10456 · localstack/localstack · GitHub 1.

Have you set the access? Can you please confirm that it’s set in the kube config file?

awslocal eks update-kubeconfig --name cluster1 && \
    kubectl config use-context arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
8

Hi @Marcel ,

Thank you for you reply.

Something in the .kube/config is:

users:
- name: rancher-desktop
  user:
    client-certificate-data: very-long-data
    client-key-data: very-long-data
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      command: aws

It seems the arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1 user doesn’t set a credential. And its command is aws. So maybe it’s the issue.
If I don’t connect to AWS, can’t I use LocalStack EKS?

9

Please try to set up the AWS configuration similar to the one in the documentation.

10

Hi @Marcel ,

Thank you. I set localstack profile and rewrite the config for kubernetes:

.aws/config

[profile localstack]
region=ap-northeast-1
output=json
endpoint_url=http://localhost:4566

.aws/credentials

[localstack]
aws_access_key_id=test
aws_secret_access_key=test

.kube/config

users:
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      - --profile
      - localstack
      command: aws

Still can’t connect to the cluster:

E0527 19:01:21.076243   22178 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials

How ever, I can get token by the command directly:

eks get-token --region ap-northeast-1 --cluster-name cluster1 --output json --profile localstack
{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2024-05-27T10:25:47Z",
        "token": "k8s-aws-v1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}
11

I’ve installed and tried to reproduce your issue with Rancher Desktop, but it’s working as expected.

Could you please try to clear the full .kube/ folder even with the cache folder?

I would also suggest doing a factory reset in the Rancher Desktop.

12

Hi @Marcel ,

I tried your suggestion. Finally, it works:

kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
k3d-cluster1-server-0   Ready    control-plane,master   25m   v1.22.6+k3s1

Really thank you very much for your help on this topic.

1 Like