Some more issues I have observed with regards to traps waiting for newcomers to Kubernetes.

Don't use `default` namespace. Just don't.

Using default namespace is easy. It just works and you have one issue less to care about, important especially when you are just starting out, isn't it? Unfortunately, it leads to horrible mess over time: 

~ λ kubectl get deployment,pods,svc,ing | wc -l
274

This is a count of just the objects we care about in a certain cluster where we used default namespace for most of deployments. While we do manage (= -l app=name = is great option!) it's not easiest to see what is happening where, especially when you are dealing with an incident.

But how I am going to prevent deploying to different namesace by mistake? If you haven't yet gone for generated manifests (which is something I also recommend), remember that you don't have to manually specify the namespace for each kubectl apply (and let's be honest - majority of us are still using kubectl apply.)

The important thing is that you can include namespace in your object metadata. The following example is a bit old (you can recognize by API version), but taken from real life where I have separate "infra" namespace housing miscallaneous but important components for running rest of the system. Accidentally deploying it to different namespace because someone typed kubectl apply -f could have quite disastrous effects, and unless you have invested time into proper Continuous Deployment pipeline that cuts you out from manual change, it's always a risk. 

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-deployment
  namespace: infra
spec:
...

Lack of parametrization in deployments

Probably one of the most annoying issues one can make for themselves is things hardcoded into container itself. Because it's a matter of when, not if, that you will have an urgent need to change something, or deploy more than one copy, and building a new container version will be either too slow, or slow you down enough to make you irritated.

Parameterizing your containers allows you operational flexibility you'll end up loving, especially with ConfigMaps/Secrets mounted as environment variables.

If you have software that needs configuration file and you don't have resources to change that at the moment, a pretty good solution is to use envsubst as part of your entrypoint script in Dockerfile.

Parameterization is also core concept of 12-factor apps, which are easier to operate in cloud native way, as they automatically pull necessary configuration from their environment. By parameterizing the whole container, often with a script as part of your Docker entrypoint, we can bring configuration from environment even to applications that don't support it natively.

envsubst is part of gettext utilities (and thus reasonably small dependency on your containers) and allows shell-style replacement of variables in files. 

# As part of your entrypoint script:

envsubst < config.template > config

# Will replace shell variable references in ${VAR} style with variables
# taken from environment

This can be utilized with envFrom in your pod specs this way: 

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dev-config
data:
  ENV: development
  DB: dev.db
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      env: dev
  template:
    metadata:
      labels:
        app: my-app
        env: dev
    spec:
      containers:
      - name: app
        image: mycorp/my-app:0.0.1
        envFrom:
        - configMapRef:
            name: dev-config

Above deployment will automatically have ENV and DB environment variables set from dev-config ConfigMap. Since ConfigMaps are namespace-scoped, you can even use standard name and use namespaces to separate them - for example, "dev" namespace might have a configmap that sets defaults specific to dev environment, while the YAML files for deployment remain the same.

Generally the more parameterized your images are, the easier it is to handle changes in environment. Tools like envsubst allow you to follow ideas of 12-factor apps without rewriting the code of your (or third-party) applications. And the less work you need to manually change something to get a configuration change applied? Always a win in my book.

Helm with Tiller (yes, really)

This might be a tiny bit controversial, as it sounds like I'm advocating against the use of Helm. Unfortunately, Helm's Tiller makes it very easy to shoot yourself in the foot, and then spend hours looking for the smoking gun.

The problem is that when you use Helm in the default way, with tiller, the manifests are applied by Tiller, not by your local kubectl. This can mean that errors in your manifests (especially dreaded space-based nesting in YAML) often won't be reported by helm install - instead you'll finally get a timeout error which doesn't tell you anything about the actual problem.

Helm 3 is going to fix this, however in the meantime if you want to use Helm, I recommend using it in so-called "tiller-less" mode. It's not best solution, but especially when you're developing your own chart, will actually give you errors if your YAML templates are incorrect. 

# Download a helm Chart
helm fetch \
  --repo https://kubernetes-charts.storage.googleapis.com \
  --untar \
  --untardir ./charts \
  --version 5.5.3 \
    prometheus
# Copy Chart's values.yaml file
cp ./charts/prometheus/values.yaml \
  ./values/prometheus.yaml
# After editing values to your liking, generate manifests:

helm template \
  --values ./values/prometheus.yaml \
  --output-dir ./manifests \
    ./charts/prometheus

# And apply them 
kubectl apply --recursive --filename ./manifests/prometheus