Cloud build error in liquibase step to enable spanner API

Question

I have setup a GCP project for build activities that support multiple GCP projects deployments, this way I have one place to manage the CI/CD and in this project no other resources APIs like spanner API are enabled.

In the CI/CD GCP projecting I'm using the cloud build and trying to configure the liquibase for spanner schema management and this is failing with error as:

ERROR: Exception Primary Reason: PERMISSION_DENIED: Cloud Spanner API has not been used in project <CLOUD_BUILD_PROJECT_ID> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/spanner.googleapis.com/overview?project=<CLOUD_BUILD_PROJECT_ID> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. li

This is the cloud build yaml file looks like:

  - id: 'download_liquibase_spanner_extension'
    name: 'gcr.io/cloud-builders/wget'
    args:
      - '-P'
      - '/workspace/louhi_ws/drivers'
      - 'https://github.com/cloudspannerecosystem/liquibase-spanner/releases/download/4.32.0/liquibase-spanner-4.32.0-all.jar'
  - id: 'run_liquibase_update'
    name: 'liquibase/liquibase:4.33.0'
    env:
      - '_TARGET_PROJECT=${_TARGET_PROJECT}'
      - '_SPANNER_INSTANCE=${_SPANNER_INSTANCE}'
      - '_SPANNER_DATABASE=${_SPANNER_DATABASE}'
    args:
      - '--changelog-file=/workspace/louhi_ws/changelog/changelog.yaml'
      - '--url=jdbc:cloudspanner:/projects/${_TARGET_PROJECT}/instances/${_SPANNER_INSTANCE}/databases/${_SPANNER_DATABASE}'
      - '--driver=com.google.cloud.spanner.jdbc.JdbcDriver'
      - '--classpath=/workspace/louhi_ws/drivers/liquibase-spanner-4.32.0-all.jar'
      - 'update'
    waitFor: ['download_liquibase_spanner_extension']

So the questions:

• Is there anyway to pass the target GCP project as input to the liquibase docker, so the target spanner API is used and billed instead of the CI/CD project billing?
• Is there any alternative to manage this flow, where CI/CD is in different project and deployments are in different projects without additional billing on the CI/CD project?

Answer 1

Cloud Spanner needs to be enabled in the build project like the error indicates. Only the project running Spanner will have any costs associated with it, but the CICD project will have its own costs associated with the APIs used for building.

You could use gcloud SDK to enable the API so no console use is required.

gcloud services list --available

gcloud services enable SERVICE_NAME

https://cloud.google.com/apis/docs/monitoring#using

Answer 2

I agree with DazWilkin and Edwrd_T_Justice that you need to enable Cloud Spanner in the target project (not the Cloud Build project, but where your Spanner instance is). It’s happening because Cloud Build is trying to interact with resources in a different project than the one where Cloud Build itself is running (the CI/CD project).

In order to fix this:

• Enable Cloud Spanner in the target project (not the Cloud Build project):

  • Go to the target project (where your Spanner instance is) and enable the Cloud Spanner API:

gcloud services enable spanner.googleapis.com --project=<TARGET_PROJECT_ID>

• Regarding your first question, you can pass the target GCP project to the Liquibase container and ensure that the Liquibase container uses the credentials from that target project.

  • You need to set up a service account to a project where the Cloud Spanner API is enabled.  After that, assign the necessary role of roles/spanner.databaseAdmin to manage the Spanner database.

  • In the Cloud Build YAML file, add this code to authenticate the build using a service account from the target project:

steps:
  # Authenticate with the service account of the target project
- id: 'authenticate_target_project'
  name: 'gcr.io/cloud-builders/gcloud'
  args:
    - 'auth'
    - 'activate-service-account'
    - '--key-file=/workspace/target-service-account.json'  # Service account key for target project

• As for managing CI/CD across multiple GCP projects without incurring additional billing in the CI/CD project, the core concept is the same: ensure that credentials for the target projects are passed along in your Cloud Build process. The billing will be tied to the resources used in the target project, as long as:

  • The service account used in the Cloud Build process is linked to the target project and not the CI/CD project.

  • You are invoking the resources (e.g., Spanner, Cloud Storage) in the target project.