Sync secret management systems into Kubernetes secret

Working with a customer who needed to manage external Kubernetes secret, we came over External Secrets.
External Secret is a Kubernetes operator that integrates external secret management systems.
It's an excellent project. You can use it with almost every Cloud provider, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault.
https://github.com/external-secrets/external-secrets
Using External Secret, you can manage from a single point of truth your Kubernetes secret. You can easily rotate them and delegate the management without providing access to your Kubernetes clusters.
We used it on Oracle Cloud Infrastructure to sync with OCI Vault.
To succeed, few steps:
Create an OCI Vault
Create a secret
Install External Secret
Create a SecretStore (2 methods):
Configure Instance Principal
Configure Auth
Create an ExternalSecret
Create an OCI Vault
If it's your first vault, create a master key, then create your secret.
 Name of the secret
 Description, optional
 Encryption key (your master encryption key)
 Secret key template: Plain-Text
 Secret Content: Your secret content.
Create ocisecret
Install External Secret Operator
Follow the step to install with Helm https://external-secrets.io/v0.4.4/guides-getting-started/
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace
Create your SecretStore
External Secret offers two ways to authenticate, using Instance principal or service account authentication.
Instance Principal
To use Instance principal, you need to create a dynamic group and IAM policy.
Configure Instance Principal
Create Dynamic Group
Instances that meet the criteria defined by any of these rules will be included in the dynamic group.
Dynamic group name: oke-kms-dyn-grp
Any {instance.compartment.id = 'ocid1.compartment.oc1..xxxxxx'}	
Any {resource.type = 'cluster',resource.compartment.id = 'ocid1.compartment.oc1..xxxxxx'}	

Use your compartment OCID.
IAM Policy
IAM policy will allow the defined dynamic group to manage secrets in your compartment.
Allow dynamic-group oke-kms-dyn-grp to manage vaults in compartment JULSILVE_ES
Allow dynamic-group oke-kms-dyn-grp to manage keys in compartment JULSILVE_ES
Allow dynamic-group oke-kms-dyn-grp to manage secret-family in compartment JULSILVE_ES

Use your compartment name.
Tips
You can SSH into your worker node to validate your instance principal configuration.
oci --auth instance_principal --region us-ashburn-1 secrets secret-bundle get --secret-id "${SECRET_OCID}"
Instance principal Secret Store
Specify the OCID of the OCI Vault and your region.
  apiVersion: external-secrets.io/v1alpha1
  kind: SecretStore
  metadata:
    name: example-instance-principal
  spec:
    provider:
      oracle:
        vault: # The vault OCID
        region: # The vault region eq: us-ashburn-1
Create your secret store
  kubectl create -f secretstore.yaml
Account Authentifcation
You must create a Kubernetes secret with the API key and the related fingerprint to use the service account method.
Create Kubernetes Secret
Generate a new API Key and fingerprint from OCI Console.
Create a Kubernetes secret with the API Key (Pem certificate) and the fingerprint.
Create Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: oracle-secret
  labels:
    type: oracle
type: Opaque
stringData:
  privateKey: |-
        -----BEGIN PRIVATE KEY-----
        insert you PEM certificate
        -----END PRIVATE KEY-----
  fingerprint: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
Create Auth Secret Store
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: secretstore-auth
spec:
  provider:
    oracle:
      vault: # The vault OCID
      region: # The vault region eq: us-ashburn-1
      auth:
        user: # The user OCID to use
        tenancy: # The tenancy OCID
        secretRef:
          privatekey:
            name: oracle-secret
            key: privateKey
          fingerprint:
            name: oracle-secret
            key: fingerprint
Once your Secret Store is created, you can confirm its validity.
$ kubectl get ss
NAME           AGE    STATUS
example-auth   5d5h   Valid
julienvault    13d    Valid
Create External Secret
Then create the external secret to sync your OCI Vault secret to your Kubernetes secret using your secret store.
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: ocisecret
  namespace: default
spec:
  refreshInterval: 0.03m
  secretStoreRef:
    kind: SecretStore
    name: julienvault
  target:
    name: k8s-secret
    creationPolicy: Owner
  data:
    - secretKey: oci-secret
      remoteRef:
        key: ocisecret
Tips
DataFrom can be used to get a secret as a JSON string and attempt to parse it.
dataFrom:
  - key: username
Data can use used to get a string, like in my example.
Secret sync as K8s secret
As result your external secret (es) is synchronized with OCI Vault.
kubectl get es ocisecret
NAME        STORE         REFRESH INTERVAL   STATUS
ocisecret   julienvault   0.03m              SecretSynced
Wrap-up


To confirm, you can query and decode the data content of the secret.
kubectl get secret k8s-secret -o jsonpath='{.data.oci-secret}'| base64 -d
tadaaa

GCP GKE VPN to on premises

Once we needed to consult an on-prem MS SQL from our PHP Lumen microservices.

Following the steps described in the following articles.

https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

Then change to false to not mask links and publish them to the firewall.

  masqLinkLocal: false

Oracle Cloud Route traffic between tenancies

With Oracle Cloud Infrastructure OCI, you can easily route traffic between tenancies using DRG and VCN attachments.

Check my blog on Oracle.com

https://blogs.oracle.com/cloud-infrastructure/post/route-traffic-between-tenancies-with-drg-and-vcn-attachment

Use GCP and gsutil to backup your archive files

I have the case of an old server, still running but already past the end of life. I needed to archive vzdump backups and virtual machine images just in case.

OS is too old to install GCP tools like gcloud and gsutil, but there is curl. So I can archive my files.

From my laptop, I can get my access token once the login is confirmed.

gcloud auth application-default login

gcloud auth application-default print-access-token

You received in response the access token to use with curl to upload the file to mybucket in Google Object Storage.

curl -v --upload-file vzdump-qemu-2210-2020_09_27-14_31_04.vma.gz -H "Authorization: Bearer access-token" https://storage.googleapis.com/mybucket/vzdump-qemu-2210-2020_09_27-14_31_04.vma.gz

Replace access-token and mybucket with your values. 

This way, I can externalize my backups and even try to provision some old machines as VM. Stay tuned.

Kubectl plugins the easy way

You know kubectl is today the Cloud console tool, the new SSH and bash and much more.

I use it every day and as always I only rely on my bash history and CTRL+R to find the last correct use.

For a long time, I read Ahmet Alp Balkan tweets about kubectl plugin but it sounds difficult.

Kubernetes the hard way, right?😁

(if you don't already follow Ahmet on Twitter)

So I was happy with my kubectl history until I assist to Paris Containers Day, especially 

Gaëlle Acas & Aurélie Vache's presentation check their video Kubectl plugin in a minute ! (in french)

Incroyable, it's super easy to create your kubectl plugin and they demonstrate in 4 steps :

1. Create a file named kubectl-myfantasticplugin
2. Make it executable, chmod +x kubectl-myfantasticplugin
3. Move it into your PATH, mv kubectl-myfantasticplugin /usr/local/bin
4. Run "kubectl myfantasticplugin"

You can create your plugin with any language and especially Go, Python, Rust, Quarkus or Bash.

Test yourself :

1. copy kubectl-halloween bash script from their Github, 
2. chmod +x kubectl-halloween 
3. sudo mv kubectl-halloween /usr/local/bin/kubectl-halloween
4.  kubectl halloween pod
5. Get your emoji

NAME                                READY   STATUS    RESTARTS   AGE

🎃 my-nginx-5d59d67564-5q4mk           1/1     Running   0          4h52m

🎃 my-nginx-5d59d67564-5shv7           1/1     Running   0          4h52m

🎃 my-nginx-5d59d67564-nwrwk           1/1     Running   0          4h52m

🎃 nginx-deployment-66b6c48dd5-m9rdt   1/1     Running   0          11d

🎃 nginx-deployment-66b6c48dd5-vzw48   1/1     Running   0          11d

It's Awesome.

https://github.com/scraly/kubectl-halloween/

The best is to come. Now you know how to create kubectl plugins. 

You can create your own, use existing ones and share yours using Krew.

You can also create your own Krew index to share your kubectl plugin in your team, project or company.

And what to do with kubectl plugins ?

Everything ! View secrets, view certificate, resources utilizations, logs, and more.

Plugins are a great way to increase your agility and productivity.

Remember your old bash script, now it's a kubectl plugin and you can share it.

Find Gaelle and Aurelie slide deck, they explain everything better. 

Thank you for the inspiration.

https://docs.google.com/presentation/d/1dfAkOGPM850fyNm8WSXGL1SWPPoW7FrcxY736iNwUJE/preview?slide=id.g9396b9860c_0_39

Official documentation

https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

And check Ahmet famous plugin

https://github.com/ahmetb/kubectx

Google Cloud Tweak Ingress and Healthcheck

Now  you have everything is on the GKE cluster, differents namespaces, deployments, products and devs.

As always locally it works on the machine but once deployed you get the terrible HTTP/502 the new Blue Screen of Death.

Why ?

Then you troubleshoot ?

    You got 502 after a 30s timeout ?

    You log to the logs /index.html and get HTTP/404 !

What's wrong ? You look to ingress configuration Nginx container, ... then you realize each products have their specificites some have no /index.html, just a response to /, other need a longer timeout to upload or process stuff and so on.

Cloud brings another layer of complexity, for this reason sometimes you need to tweak backend-services and health-checks.

By default backend-services (loadbalancer) have a 30s timeout default.

You can list them and find you backend-services rules

gcloud compute backend-services list

Sometimes it's easier from the console to get the loadbalancer then the backend service you need.

Then you can check with describe

gcloud compute backend-services describe k8s-be-30080--9747qarggg396bf0 --global

Then you can update your timeout or any other settings

gcloud compute backend-services update k8s-be-30080--9747qarggg396bf0 --global --timeout=600

Take a coffee to give time to apply and Bingo your HTTP/502 disappear.

Well this one.

You can also tweak healthcheck

From the console find the healthcheck you need.

You can also list them 

gcloud compute health-checks list --global

Then describe to control

gcloud compute health-checks describe k8s-be-30569--9747df6bftswwq5c396bf0

Update the healthcheck to your needs

gcloud compute health-checks update http k8s-be-30569--9747df6bftswwq5c396bf0 --request-path=/

Now you managed a second HTTP/502 error.

Congratulations

What's next ?

Automate backup Google Cloud CoudSQL

Google Cloud offers automatic backups but theses backups are bind to the instance.

You can only retain 7 and export them. Also if you need to restore just one database or table you will have to restore all the data of the instance. Finally and more important your business needs may require more frequent backups a smaller RPO.

Solution is automate export of your instances. This way you can choose the tables or databases to export and their frequency.

To do it I what to use Cloud Scheduler, Pub/Sub, Cloud Functions and Cloud Storage.

Based on the following blog I made several attempts

https://revolgy.com/blog/how-to-automated-long-term-cloud-sql-backups-step-by-step-guide/

But somethings were missing :

1 - IAM, the permission to export in is Cloud SQL Viewer role and not in Cloud SQL Client role.

You may create a custom role or grant to use Cloud SQL Viewer.

2 - ExportContext.databases API is different if you use Mysql or PostgreSQL instance.

Databases to be exported.
MySQL instances: If fileType is SQL and no database is specified, all databases are exported, except for the mysql system database. If fileType is CSV, you can specify one database, either by using this property or by using the csvExportOptions.selectQuery property, which takes precedence over this property.
PostgreSQL instances: You must specify one database to be exported. If fileType is CSV, this database must match the one specified in the csvExportOptions.selectQuery property.

So if you use Mysql, you many not specify a database name to export all of them.

But if you use Pgsql you have to specify a database name.

https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1beta4/operations#ExportContext

This way I use two different schedulers, one for each instance with different payloads, same Pub/Sub topic and same Function.

Finally in the API it's databaseS and not database 😀 cost me some time to figure out my mistake.

Now backups are automated, exported to a bucket with a life cycle.

Production is safe and each dev can download dev db anytime.

Now I need to update the function to parse database of the instance and try a restoration 😄

Thank you