This is a text-only version of the following page on https://raymii.org:
---
Title       : 	Create Kubernetes user restricted to one namespace with resource limits
Author      : 	Remy van Elst
Date        : 	29-07-2024 04:39
URL         : 	https://raymii.org/s/tutorials/Create_Kubernetes_user_restricted_to_one_namespace_with_resource_limits.html
Format      : 	Markdown/HTML
---



This guide shows you how to use Role-based access control (RBAC) to create a user account that only has rights for one specific namespace. I'll also show you how to limit the resource usage of that  `Namespace`.  Last but not least, I'll also show you how to create a `kubeconfig` file for that specific user.

<p class="ad"> <!--<b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below. It means the world to me if you  show your appreciation and you'll help pay the server costs:</b><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">GitHub Sponsorship</a><br><br> <a href="https://pcbway.com/g/e7yQRg">PCBWay referral link (You get $5, I get $20 after you've placed an order)</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">Digital Ocea referral link  ($200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!)</a><br><br>--> </p>


You can use this guide to set up a limited resource for one team for example,
to make sure they cannot mess up other parts of the cluster or slow down
other stuff due to claiming too many resources. My sysadmin / devops
experience tells me that no matter what, if there are no limits or guard
rails, developers (especially web and PHP developers, they are the worst
kind, see [this article for example, a hacked cluster and all the
web-developers did was scale up due to "performance". An actually skilled
developer would look into the problem, do profiling and fix pain points
before scaling out (and have their basic security in order).](https://it-notes.dragas.net/2024/07/04/from-cloud-chaos-to-freebsd-efficiency/#lessons-learned))
will mess up operations, no matter how well intended. Better to protect them
against themselves than to be woken up in the middle of the night.


![dashboard](/s/inc/img/k3s-namespace-2.png)

> The Kubernetes Dashboard for this one user with permissions in only that namespace

![resource quotas](/s/inc/img/k3s-namespace-1.png)

> Resource limits in the namespace


To read all my [Kubernetes posts, click here](/s/tags/kubernetes.html). I'm
using Kubernetes / k3s version ` v1.30.2+k3s1` and for the purposes of this
guide I assume you have `kubectl` set up and working with an admin user.

The [official documentation]
(https://web.archive.org/web/20240729082142/https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
is a great resource which explains this stuff in more detail. This is a
practical guide for a single purpose (namely to create a namespace for a user
with resource limits and denying the user access to further namespaces). 

Make sure to also read up on [privilege escalation]
(https://web.archive.org/web/20240729102458/https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping)
in Kubernetes.

A few terms require some more explanation: 

- `Role`: A role contains rules that represent a set of permissions. A role is
  used to grant access to resources within a namespace. Permissions are
  purely additive (there are no "deny" rules). A `Role` is always Namespaced
  (as opposed to a `ClusterRole`)
- `RoleBinding`: A role binding is used to grant the permissions defined in a
  role to a user or set of users. It holds a list of subjects (users, groups,
  or service accounts), and a reference to the role being granted
- `Service Account`: account meant for processes, which run in pods
  (something that talks to Kubernetes). Or in our case, `kubeconfig`


Create a folder in which we will put all the `yaml` files:

    mkdir user-demo
    cd user-demo


### Creating a Namespace with a `ResourceQuota`

Create a file for your namespace:

    vim namespace.yml
    
Contents:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: user-demo

Apply it:

    kubectl apply -f namespace.yml
    
Output:

    namespace/user-demo created


Create a file with the resource limits for this namespace:
 
    vim resource-quota.yml
    
Contents:

    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: user-demo-quota
      namespace: user-demo
    spec:
      hard:
      requests.cpu: "1"
      requests.memory: 1Gi
      limits.cpu: "2"
      limits.memory: 2Gi
    
Apply the quota:

    kubectl apply -f resource-quota.yml

The above  `ResourceQuota` places these requirements on the `user-demo` namespace:

- For every Pod in the namespace, each container must have a memory request,
  memory limit, cpu request, and cpu limit.
- The memory request total for all Pods in that namespace must not exceed 800
  MiB. The memory limit total for all Pods in that namespace must not exceed
  950 MiB.
- The CPU request total for all Pods in that namespace must not exceed 1 cpu.
- The CPU limit total for all Pods in that namespace must not exceed 2 cpu.

This means that in your `Deployment` yaml files you must make sure there are
resource limits. Below is an example.


### Create a Deployment with resource limits

We'll create two deployments, one of which will succeed and one of which will
fail due to resource limitations.

Create a file for our test deployments:
  
    vim deployment.yml
    
Contents:

      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: echo-1
        namespace: user-demo
      spec:
        replicas: 1
        selector:
        matchLabels:
          app: echo-1
        template:
        metadata:
          labels:
          app: echo-1
        spec:
          containers:
          - name: echo-1
            image: ealen/echo-server:latest
            ports:
            - containerPort: 80
            livenessProbe:
            httpGet:
              path: /?echo_code=200
              port: 80
            readinessProbe:
            httpGet:
              path: /?echo_code=200
              port: 80
            resources:
            limits:
              cpu: 800m
              memory: 600Mi
            requests:
              cpu: 600m
              memory: 400Mi
      ---

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: echo-2
        namespace: user-demo
      spec:
        replicas: 1
        selector:
        matchLabels:
          app: echo-2
        template:
        metadata:
          labels:
          app: echo-2
        spec:
          containers:
          - name: echo-2
            image: ealen/echo-server:latest
            ports:
            - containerPort: 80
            livenessProbe:
            httpGet:
              path: /?echo_code=200
              port: 80
            readinessProbe:
            httpGet:
              path: /?echo_code=200
              port: 80
            resources:
            limits:
              cpu: 800m
              memory: 600Mi
            requests:
              cpu: 600m
              memory: 400Mi

You might wonder what `cpu:600m` means. In our case the limit is `1` and this sort of means, `0.6`. I've used the [echo-server](/s/tutorials/Kubernetes_k3s_Ingress_for_different_domains_like_virtual_hosts.html)
in earlier guides.
  
Apply the file:

    kubectl  apply -f deployment.yml
  
Output    

    deployment.apps/echo-1 created
    deployment.apps/echo-2 created
  
If you query the status of the deployments you'll see the first being created
successfully, the second is not:

    kubectl -n user-demo get deployments
  
Output:   
  
    NAME     READY   UP-TO-DATE   AVAILABLE   AGE
    echo-1   1/1     1            1           54s
    echo-2   0/1     0            0           54s

To figure out why, query the `Deployment`:

    kubectl -n user-demo describe deployment echo-2

Output, trimmed:

    Conditions:
      Type             Status  Reason
      ----             ------  ------
      Progressing      True    NewReplicaSetCreated
      Available        False   MinimumReplicasUnavailable
      ReplicaFailure   True    FailedCreate



Something is wrong with our `ReplicaSet`. Let's query that:

    $ kubectl -n user-demo get replicaset
  
Output:   
  
    NAME                DESIRED   CURRENT   READY   AGE
    echo-1-777547b855   1         1         1       2m11s
    echo-2-6fbd7564f7   1         0         0       2m11s


Query the second `ReplicaSet`:

    kubectl -n user-demo describe replicaset echo-2-6fbd7564f7
  
Output, trimmed:

    Name:           echo-2-6fbd7564f7
    Namespace:      user-demo
    [...]
    Replicas:       0 current / 1 desired
    Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
    Pod Template:
      [...]
      Limits:
        cpu:     800m
        memory:  600Mi
      Requests:
        cpu:         600m
        memory:      400Mi
      [...]
    Conditions:
      Type             Status  Reason
      ----             ------  ------
      ReplicaFailure   True    FailedCreate
    Events:
      Type     Reason        Age                  From                   Message
      ----     ------        ----                 ----                   -------
      Warning  FailedCreate  2m31s                replicaset-controller  Error creating: pods "echo-2-6fbd7564f7-bbnld" is forbidden: exceeded quota: user-demo-quota, requested: requests.cpu=600m, used: requests.cpu=600m, limited: requests.cpu=1
      Warning  FailedCreate  2m31s                replicaset-controller  Error creating: pods "echo-2-6fbd7564f7-vhfdk" is forbidden: exceeded quota: user-demo-quota, requested: requests.cpu=600m, used: requests.cpu=600m, limited: requests.cpu=1
      Warning  FailedCreate  2m31s                replicaset-controller  Error creating: pods "echo-2-6fbd7564f7-6qpht" is forbidden: exceeded quota: user-demo-quota, requested: requests.cpu=600m, used: requests.cpu=600m, limited: requests.cpu=1
      [...]


The error is quite clear, we're out of resources! Exactly what we want. There is more info on resource limits in the documentation, you can also limit [the amount of Pods](https://web.archive.org/web/20240727083328/https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-pod-namespace/) and [here you can find other limits, for Storage or Ingress or Services](https://web.archive.org/web/20240729085730/https://kubernetes.io/docs/concepts/policy/resource-quotas/) for example.

Here is the example resource quota extended with storage, services and ingresses:

    spec:
      hard:
      requests.cpu: "1"
      requests.memory: "1Gi"
      limits.cpu: "2"
      limits.memory: "2Gi"
      services.loadbalancers: "2"
      count/ingresses.networking.k8s.io: "2"
      persistentvolumeclaims: "4"
      requests.storage: "8Gi"

Then, when creating more `Loadbalancers` or `Ingresses`, you will receive an
error like so:

    Error from server (Forbidden): error when creating "deployment.yml": services "echo-2-service-2" is forbidden: exceeded quota: user-demo-quota, requested: services.loadbalancers=1, used: services.loadbalancers=2, limited: services.loadbalancers=2
    Error from server (Forbidden): error when creating "deployment.yml": ingresses.networking.k8s.io "echo-2-ingress-2" is forbidden: exceeded quota: user-demo-quota, requested: count/ingresses.networking.k8s.io=1, used: count/ingresses.networking.k8s.io=2, limited: count/ingresses.networking.k8s.io=2

Delete the test deployments to free up our resources:

    kubectl -n user-demo delete -f deployment.yml

### Creating a user restricted to one namespace

Now that we have a namespace with resource limits, we can create a user bound to that namespace. 

As we stated earlier, you need a few pieces, not "just" a user. Start with the
`ServiceAccount`, this is comparable to your "User". I'm assuming your
namespace `user-demo` is already created, if not, see the top of this page.
I'm going to use the term `User` interchangeably with `ServiceAccount` in the
rest of this guide.

Create a file for the user:

    vim user1-servivceaccount.yml
    
Contents:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: user1
      namespace: user-demo

Apply the file to create the service account:

    kubectl apply -f user1-serviceaccount.yml
    
Output:

    serviceaccount/user1 created

Next up is the `Role`, this file describes the specific permissions for the
`Role`. The role will be later bound to the `ServiceAccount`. That is a
separate process because one role can be bound to more than one user.

    vim namespace-admin-role.yml
  
Contents:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: namespace-admin
      namespace: user-demo
      rules:
    - apiGroups: ["", "extensions", "apps"]
      resources: ["*"]
      verbs: ["*"]
    - apiGroups: ["batch"]
      resources:
      - jobs
      - cronjobs
      verbs: ["*"]

This role grants permissions to perform almost all actions within the
namespace an Admin User would do.

You might want to have a more limited profile, for example, just a
`Deployment` admin user:

    [...]
      name: deployment-admin
    rules:
    - apiGroups: ["", "extensions", "apps"]
      resources: ["deployments", "replicasets", "pods", "services", "ingresses"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] 
      

Apply the file:   

    kubectl apply -f namespace-admin-role.yml
  
Output:

    role.rbac.authorization.k8s.io/namespace-admin created
  
Last step in user / role creation is the `RoleBinding`. This is what couples the `Role` to the User.

    vim namespace-admin-rolebinding.yml
  
Contents:

    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: namespace-admin-rolebinding
      namespace: user-demo
    subjects:
    - kind: ServiceAccount
      name: user1
      namespace: user-demo
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: namespace-admin

You can have multiple `subjects` but for our example we only need one. Apply the file:

    kubectl  apply -f namespace-admin-rolebinding.yml

Output:

    rolebinding.rbac.authorization.k8s.io/namespace-admin-rolebinding created
  
One more step remaining to actually use the new user.   

### Creating a kubeconfig file for the user

To use the new user with permissions, the last step is to create a
`kubeconfig` file. You can give that file to someone and they then can use
`kubectl` within that namespace, or create a token for the dashboard for
example.

Start by creating a long-lived token (10 years) for the user. Update the
duration to suite your needs.

    kubectl create token user1 -n user-demo --duration=87600h
    
Output:

    eyJhbGciOiJ[...]VwRTkA

Create a `user1-kubeconfig` file:

    vim user1-kubeconfig

Contents:

    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
      certificate-authority-data: <base64-encoded-CA-cert> 
      server: https://<your-cluster-endpoint> 
      name: default
    contexts:
    - context:
      cluster: default
      namespace: user-demo
      user: user1
      name: user1-context
    current-context: user1-context
    users:
    - name: user1
      user: 
      token: <your-token>  

The `cluster.server` can be found using this command:

    kubectl cluster-info
  
Output:

    Kubernetes control plane is running at https://192.0.2.60:6443

The token you just created should be pasted into `user.token`. The last
part, `certificate-authority-data` can be (from Kubernetes 1.24 and up)
queried from a `configMap`:

    kubectl get configmap kube-root-ca.crt -n kube-public -o jsonpath="{['data']['ca\\.crt']}" | base64 -w 0
  
Output:   
  
    LS0[...]tLS0K

You can now use this file with `kubectl` by providing the `--kubeconfig` parameter:

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo get pods

If you try to list another namespace for which the account has no permissions, you will receive errors:

     kubectl  --kubeconfig ./kubeconfig.yaml -n default get all
  
Output:

    Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:user-demo:user1" cannot list resource "pods" in API group "" in the namespace "default"
    Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:user-demo:user1" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"

You can also create the deployment (from earlier in this article) as this user:

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo apply -f deployment.yml

Output:
  
    deployment.apps/echo-1 created
    deployment.apps/echo-2 created

Querying, deleting and scaling works as you would expect:

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo get deployment
  
Output:

    NAME     READY   UP-TO-DATE   AVAILABLE   AGE
    echo-1   1/1     1            1           40s
    echo-2   0/1     0            0           40s

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo get pods
        
Output:

    NAME                      READY   STATUS    RESTARTS   AGE
    echo-1-777547b855-ftzzk   1/1     Running   0          41s

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo delete deployment echo-2
  
Output:

    deployment.apps "echo-2" deleted

    kubectl  --kubeconfig ./kubeconfig.yaml -n user-demo scale deployment echo-1 --replicas 2
  
Output:

    deployment.apps/echo-1 scaled

You can use the token to login to the Kubernetes Dashboard as well, but you
cannot port-forward:
  
    kubectl  --kubeconfig ./kubeconfig.yaml -n kubernetes-dashboard  port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
  
Output:

    Error from server (Forbidden): services "kubernetes-dashboard-kong-proxy" is forbidden: User "system:serviceaccount:user-demo:user1" cannot get resource "services" in API group "" in the namespace "kubernetes-dashboard"

This is expected because we do not have permissions in any other namespace.
Set up an Ingress or NodePort for the dashboard in a trusted environment and
you can use the token to login.


---

License:
All the text on this website is free as in freedom unless stated otherwise. 
This means you can use it in any way you want, you can copy it, change it 
the way you like and republish it, as long as you release the (modified) 
content under the same license to give others the same freedoms you've got 
and place my name and a link to this site with the article as source.

This site uses Google Analytics for statistics and Google Adwords for 
advertisements. You are tracked and Google knows everything about you. 
Use an adblocker like ublock-origin if you don't want it.

All the code on this website is licensed under the GNU GPL v3 license 
unless already licensed under a license which does not allows this form 
of licensing or if another license is stated on that page / in that software:

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

Just to be clear, the information on this website is for meant for educational 
purposes and you use it at your own risk. I do not take responsibility if you 
screw something up. Use common sense, do not 'rm -rf /' as root for example. 
If you have any questions then do not hesitate to contact me.

See https://raymii.org/s/static/About.html for details.