Google Cloud Tweak Ingress and Healthcheck

Now  you have everything is on the GKE cluster, differents namespaces, deployments, products and devs.

As always locally it works on the machine but once deployed you get the terrible HTTP/502 the new Blue Screen of Death.

Why ?

Then you troubleshoot ?

    You got 502 after a 30s timeout ?

    You log to the logs /index.html and get HTTP/404 !

What's wrong ? You look to ingress configuration Nginx container, ... then you realize each products have their specificites some have no /index.html, just a response to /, other need a longer timeout to upload or process stuff and so on.

Cloud brings another layer of complexity, for this reason sometimes you need to tweak backend-services and health-checks.

By default backend-services (loadbalancer) have a 30s timeout default.

You can list them and find you backend-services rules

gcloud compute backend-services list

Sometimes it's easier from the console to get the loadbalancer then the backend service you need.

Then you can check with describe

gcloud compute backend-services describe k8s-be-30080--9747qarggg396bf0 --global

Then you can update your timeout or any other settings

gcloud compute backend-services update k8s-be-30080--9747qarggg396bf0 --global --timeout=600

Take a coffee to give time to apply and Bingo your HTTP/502 disappear.

Well this one.

You can also tweak healthcheck

From the console find the healthcheck you need.

You can also list them 

gcloud compute health-checks list --global

Then describe to control

gcloud compute health-checks describe k8s-be-30569--9747df6bftswwq5c396bf0

Update the healthcheck to your needs

gcloud compute health-checks update http k8s-be-30569--9747df6bftswwq5c396bf0 --request-path=/

Now you managed a second HTTP/502 error.

Congratulations

What's next ?

Automate backup Google Cloud CoudSQL

Google Cloud offers automatic backups but theses backups are bind to the instance.

You can only retain 7 and export them. Also if you need to restore just one database or table you will have to restore all the data of the instance. Finally and more important your business needs may require more frequent backups a smaller RPO.

Solution is automate export of your instances. This way you can choose the tables or databases to export and their frequency.

To do it I what to use Cloud Scheduler, Pub/Sub, Cloud Functions and Cloud Storage.

Based on the following blog I made several attempts

https://revolgy.com/blog/how-to-automated-long-term-cloud-sql-backups-step-by-step-guide/

But somethings were missing :

1 - IAM, the permission to export in is Cloud SQL Viewer role and not in Cloud SQL Client role.

You may create a custom role or grant to use Cloud SQL Viewer.

2 - ExportContext.databases API is different if you use Mysql or PostgreSQL instance.

Databases to be exported.
MySQL instances: If fileType is SQL and no database is specified, all databases are exported, except for the mysql system database. If fileType is CSV, you can specify one database, either by using this property or by using the csvExportOptions.selectQuery property, which takes precedence over this property.
PostgreSQL instances: You must specify one database to be exported. If fileType is CSV, this database must match the one specified in the csvExportOptions.selectQuery property.

So if you use Mysql, you many not specify a database name to export all of them.

But if you use Pgsql you have to specify a database name.

https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1beta4/operations#ExportContext

This way I use two different schedulers, one for each instance with different payloads, same Pub/Sub topic and same Function.

Finally in the API it's databaseS and not database 😀 cost me some time to figure out my mistake.

Now backups are automated, exported to a bucket with a life cycle.

Production is safe and each dev can download dev db anytime.

Now I need to update the function to parse database of the instance and try a restoration 😄

Thank you