What does kubectl even do?

kubectl is just one means of accessing your k8s cluster API in the k8s architecture (copied below). There is a web ui for the same API, and of course you can hit it through rest requests. kubectl is a good client for the API, so feel free to use it often.

Essentially you use this API to specify compute resources:

  • deployments, which outline pods of containers to run or migrate.
  • services, which put an abstract endpoint in front of some matching query for pods of containers that may go down or move.
  • jobs which run pods of containers that aren’t long lived.
  • You can also run pods themselves or a few other groupings of pods.

You can also establish access rules, resources limits, and run CI pipelines on these compute resources. We’ll look into some of this in a bit.

k8s architecture diagram

I recommend poking around the the k8s swagger spec to get a better feel for the API.

Who runs kubectl? I can’t just have everybody willy-nilly creating and taking down new services in production. Should this access be admin only?

On the one hand, you have to use the API to manage deployments, services, and run jobs in production. You need to audit access and changes to all your production resources and keep them stable against your SLA’s.

On the other hand, you still want to allow debugging and inspection for these production resources for developers and others, and you want to allow ad-hoc testing and research oriented compute usage in new non-production resources for many people. You want to limit the budget that can be spent on non-production compute resources. And, you want to completely ensure that non-production pods, etc., have no impact on the uptime and stability of production resources.

You accomplish stable deployments, services, and jobs by only granting access to those specific resources to your CI workers and a restricted group of admins, and no one else.

You keep yaml definitions of the resources in git repos and update them through a normal pull request and continuous integration flow running on your permitted CI workers in your k8s cluster.

You can give more general groups of users rights to inspect, without changing, your production resources. Additionally you can run logging and monitoring tools in your k8s cluster like kibana/elasticsearch and influxdb/grafana to let an even broader group of users get most of the same information without using the k8s API. These tools are also just convenient.

You can create more roles to allow some users to create pods or other resources on the fly to develop, test, or research using non-production resources.

We’ll look into CI on k8s, access, and types of users below.

Running your Deployment, Service, Job, and Container definitions.

Keep your production deployments, services, jobs, container definitions, and everything else you administer for your k8s cluster, and that needs auditing and stability, in source control, and deploy directly from source control. Other things can run outside of source control and CI.

The k8s documentors refer to this as a best practice several times throughout their docs. We can find github repos with yaml resource definitions committed in the k8s guestbook example project and the k8s mysql-galera example project, along with others. In config best practices we find:

Configuration files should be stored in version control before being pushed to the cluster. This allows a configuration to be quickly rolled back if needed, and will aid with cluster re-creation and restoration if necessary.

Use kubectl create -f where possible. This looks for config objects in all .yaml, .yml, and .json files in and passes them to create.

We can find the same instructions elaborated in managing deployments, which even references Martin Fowler’s infrastructure as code.

kubectl conventions adds a handy tip about building and tagging our container images, in addition to repeating ideas about “reusable scripts” and “infrastructure as code”:

Always tag your image with a version-specific tag and don’t move that tag to a new version. For example, use :v1234, v1.2.3, r03062016-1-4, rather than :latest (see Best Practices for Configuration for more information.)

Types of Users

We can see some types of roles the k8s devs imagined would exist in k8s types of access:


Each of these can act as normal users or attackers.

External Users: People who are accessing applications running on K8s (e.g. a web site served by webserver running in a container on K8s), but who do not have K8s API access.

K8s Users: People who access the K8s API (e.g. create K8s API objects like Pods)

K8s Project Admins: People who manage access for some K8s Users

K8s Cluster Admins: People who control the machines, networks, or binaries that make up a K8s cluster.

K8s Admin means K8s Cluster Admins and K8s Project Admins taken together.

We can see some more details about day to day responsibilities in k8s security roles:


write pod specs.
making some of their own images, and using some "community" docker images
know which pods need to talk to which other pods
decide which pods should share files with other pods, and which should not.
reason about application level security, such as containing the effects of a local-file-read exploit in a webserver pod.
do not often reason about operating system or organizational security.
are not necessarily comfortable reasoning about the security properties of a system at the level of detail of Linux Capabilities, SELinux, AppArmor, etc.

Project Admins:

allocate identity and roles within a namespace
reason about organizational security within a namespace
don't give a developer permissions that are not needed for role.
protect files on shared storage from unnecessary cross-team access
are less focused about application security


are less focused on application security. Focused on operating system security.
protect the node from bad actors in containers, and properly-configured innocent containers from bad actors in other containers.
comfortable reasoning about the security properties of a system at the level of detail of Linux Capabilities, SELinux, AppArmor, etc.
decides who can use which Linux Capabilities, run privileged containers, use hostPath, etc.
e.g. a team that manages Ceph or a mysql server might be trusted to have raw access to storage devices in some organizations, but teams that develop the applications at higher layers would not.

We can also create specific accounts for services like CI workers or any other programmatic user.

Restricting k8s API Access

K8s has an access scheme of API authentication, authorization, and even another layer of specific controls. I’ll just copy another image from their docs that makes this clear. k8s auth diagram

You can bring your own authentication to let k8s know about the users you have. And, you can use files, k8s internal groups, or webhooks, to permit or deny both humans and programmatic users to invoke operations like get, list, create, update, patch, watch, and delete on API resource types such as the following from the k8s API resource definitions:

  • v1.Pod
  • v1.Service
  • v1.ServiceList
  • v1.Node
  • v1.Event
  • v1.ResourceQuota
  • v1.Namespace
  • v1.Secret
  • v1.ServiceAccount
  • v1.PersistentVolume
  • v1.PersistentVolumeClaim
  • v1.DeleteOptions
  • v1.ComponentStatus
  • v1.ConfigMap

You can also establish resource quotas for resource usage.

Complete Audit Logs

There is a pending github issue, k8s issue #22, to produce an audit log of all API accesses, but versioning and a CI pipeline should be adequate audit tools in the meantime.

How do we run kubernetes on our server racks?

I guess start with: http://kubernetes.io/docs/getting-started-guides/#on-premises-vms



How do we setup dns?

Pods and services will have their own IP addresses. kube-dns runs in kubernetes, and resolves service names to loadbalancers for the services, and pod names to the IP addresses for the pods.

K8s has a very thought out view of API’s

Check it out: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md