Use GCP and gsutil to backup your archive files

I have the case of an old server, still running but already past the end of life. I needed to archive vzdump backups and virtual machine images just in case.

OS is too old to install GCP tools like gcloud and gsutil, but there is curl. So I can archive my files.

From my laptop, I can get my access token once the login is confirmed.

gcloud auth application-default login

gcloud auth application-default print-access-token

You received in response the access token to use with curl to upload the file to mybucket in Google Object Storage.

curl -v --upload-file vzdump-qemu-2210-2020_09_27-14_31_04.vma.gz -H "Authorization: Bearer access-token" https://storage.googleapis.com/mybucket/vzdump-qemu-2210-2020_09_27-14_31_04.vma.gz

Replace access-token and mybucket with your values. 

This way, I can externalize my backups and even try to provision some old machines as VM. Stay tuned.

Google Cloud Tweak Ingress and Healthcheck

Now  you have everything is on the GKE cluster, differents namespaces, deployments, products and devs.

As always locally it works on the machine but once deployed you get the terrible HTTP/502 the new Blue Screen of Death.

Why ?

Then you troubleshoot ?

    You got 502 after a 30s timeout ?

    You log to the logs /index.html and get HTTP/404 !

What's wrong ? You look to ingress configuration Nginx container, ... then you realize each products have their specificites some have no /index.html, just a response to /, other need a longer timeout to upload or process stuff and so on.

Cloud brings another layer of complexity, for this reason sometimes you need to tweak backend-services and health-checks.

By default backend-services (loadbalancer) have a 30s timeout default.

You can list them and find you backend-services rules

gcloud compute backend-services list

Sometimes it's easier from the console to get the loadbalancer then the backend service you need.

Then you can check with describe

gcloud compute backend-services describe k8s-be-30080--9747qarggg396bf0 --global

Then you can update your timeout or any other settings

gcloud compute backend-services update k8s-be-30080--9747qarggg396bf0 --global --timeout=600

Take a coffee to give time to apply and Bingo your HTTP/502 disappear.

Well this one.

You can also tweak healthcheck

From the console find the healthcheck you need.

You can also list them 

gcloud compute health-checks list --global

Then describe to control

gcloud compute health-checks describe k8s-be-30569--9747df6bftswwq5c396bf0

Update the healthcheck to your needs

gcloud compute health-checks update http k8s-be-30569--9747df6bftswwq5c396bf0 --request-path=/

Now you managed a second HTTP/502 error.

Congratulations

What's next ?

Automate backup Google Cloud CoudSQL

Google Cloud offers automatic backups but theses backups are bind to the instance.

You can only retain 7 and export them. Also if you need to restore just one database or table you will have to restore all the data of the instance. Finally and more important your business needs may require more frequent backups a smaller RPO.

Solution is automate export of your instances. This way you can choose the tables or databases to export and their frequency.

To do it I what to use Cloud Scheduler, Pub/Sub, Cloud Functions and Cloud Storage.

Based on the following blog I made several attempts

https://revolgy.com/blog/how-to-automated-long-term-cloud-sql-backups-step-by-step-guide/

But somethings were missing :

1 - IAM, the permission to export in is Cloud SQL Viewer role and not in Cloud SQL Client role.

You may create a custom role or grant to use Cloud SQL Viewer.

2 - ExportContext.databases API is different if you use Mysql or PostgreSQL instance.

Databases to be exported.
MySQL instances: If fileType is SQL and no database is specified, all databases are exported, except for the mysql system database. If fileType is CSV, you can specify one database, either by using this property or by using the csvExportOptions.selectQuery property, which takes precedence over this property.
PostgreSQL instances: You must specify one database to be exported. If fileType is CSV, this database must match the one specified in the csvExportOptions.selectQuery property.

So if you use Mysql, you many not specify a database name to export all of them.

But if you use Pgsql you have to specify a database name.

https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1beta4/operations#ExportContext

This way I use two different schedulers, one for each instance with different payloads, same Pub/Sub topic and same Function.

Finally in the API it's databaseS and not database 😀 cost me some time to figure out my mistake.

Now backups are automated, exported to a bucket with a life cycle.

Production is safe and each dev can download dev db anytime.

Now I need to update the function to parse database of the instance and try a restoration 😄

Thank you

Migrate DNS to Google Cloud Platform

Just done it, super easy to migrate your DNS service to GCP.

From your account, I use Cloud shell session directly.

Replace silverston.fr and silverston by your domain

Create your new zone :

gcloud dns managed-zones create --dns-name="silverston.fr." --description="My awesome domain" "silverston" 

Import your zone

gcloud dns record-sets import -z=silverston --zone-file-format silverston.fr.txt --delete-all-existing 

--delete-all-existing is necessary to delete existing NS records and use Google instead.

Get your GCP NS servers :

gcloud dns managed-zones describe silverston

You will get your NS servers, for example :

nameServers: 

- ns-cloud-a1.googledomains.com.
- ns-cloud-a2.googledomains.com.
- ns-cloud-a3.googledomains.com.
- ns-cloud-a4.googledomains.com.

gcloud dns managed-zones describe examplezonename

Update your NS in your current registar to use googledomains (use the servers you get in the previous step)

And you're done.
Control DNS propagation using :

watch dig +short NS silverston.fr

source: https://cloud.google.com/dns/docs/migrating

Stackdriver installer agent sur version non supporte de Ubuntu

Je teste GCP, l'objet d'un futur post ?
Et je teste le monitoring avec Stackdriver mais de chance stackdriver ne supporte pas Ubuntu 17.04.
Impossible d'installer l'agent a moins de modifier le script d'installation : stack-install.sh

En trichant un peu, remplacer la commande lsb_release -sc par la derniere version supportee : xenial
  -local CODENAME="$(lsb_release -sc)"
  +local CODENAME="xenial"

Et le tour est joue.

Creer de la memoire swap sur des petites instances cloud

Si comme moi vous utilisez des petites instances Cloud chez Google Cloud, Digital Ocean ou autres, des petits servers a $5 avec 512Mo de ram et pas de swap. Parfois il ne reste plus assez de memoire disponible.
Par example, c'est le cas sur mon serveur : Asterisk, LAMP.

Dans ce cas le plus simple est de creer un fichier swap qui permet d'augmenter les ressources de votre serveur sans en augmenter le cout et qui en plus vous permet de profiter des performances de votre disque dur "virtuel", du SSD presque comme de la RAM.

• fallocate -l 2G /swapfile
• chmod 600 /swapfile
• mkswap /swapfile
• editer votre /etc/fstab
• /swapfile    swap    swap    defaults    0 0
• swapon -av

Plus rapide que de faire un dd, j'utilise fallocate pour creer un fichier swap de 2G. Puis swapon pour activer la swap et le tour est joue (comme ca vous etes sur que l'entree dans fstab est correcte).

Fini l'alerte concernant la swap et surtout un peu plus de memoire plus d'appels qui se coupe, enfin possible d'upgrader le serveur, ...

Envoyer des emails depuis instance Google Cloud Platform avec Mailgun

Premiers pas sur Google Cloud Platform, GCP 
Google bloque tous les emails sortant passant par le port TCP/25 (SMTP) et invite a utiliser un tiers :

• SendGrid, Mailgun, ou Mailjet
• Google Apps domain
• Serveur email prive 

Personnellement j'ai choisi Mailgun : gratuit - 30 000 emails envoyé par mois. Il suffit d'un compte Google.

Pour utiliser Mailgun, il faut realiser deux configuration comme pre-requis.

Configurer Postfix

Configurer Postfix de la façon suivante :

relayhost = [smtp.mailgun.org]:2525
smtp_tls_security_level = may
smtpd_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:postmaster@mondomain.com:monpassword
smtp_sasl_security_options = noanonymous
smtp_sasl_auth_enable = yes

Le nom d'utilisateur et son mot de passe son disponible sur le Dashboard de Mailgun

 

Valider son domaine : configurer son DNS

Mailgun a besoin de valider le domaine pour envoyer et même recevoir les emails, pour valider le domaine, créer les enregistrements suivants :

• TXT     “v=spf1 include:mailgun.org ~all”     SPF (Required)
• TXT     copier le DKIM (disponible sur le dashboard)
• CNAME     “mailgun.org”     Tracking (pour effectuer un suivi, optionel)

Par exemple :

• Name : @, type:  TXT valeur : "v=spf1 include:mailgun.org ~all"
• Name : mx._domainkey, type : TXT, valeur : "k=rsa; p=...MIGfMA0GCSqGS"

Attendre la réplication DNS et le domaine doit passer comme "Valider, en vert"

Une fois réaliser et valider ces deux pre-requis, le compte Mailgun est fonctionnel et permet d'envoyer des emails depuis votre instance Google Cloud.

Enfin Mailgun permet également de recevoir des emails, de réaliser un suivi, de créer des campagnes et propose de nombreuses API.
De facon, on realise ces premiers pas dans le Cloud.