Which service categories make up Google Cloud Platform?
KnownRevise
Google Cloud Platform (GCP) is Google’s public cloud platform for hosting applications, storing and analyzing data, running containers and virtual machines, connecting networks, and operating production systems. Its services are organized into major categories such as compute, storage, databases, networking, messaging, analytics, AI, security, and DevOps.
Read full answerHide full answer
Applies toGoogle Cloud Platform current services and Google Cloud CLI current stable release
Resource Hierarchy and Identity
GCP uses a hierarchical resource model:
Organization: The top-level container for a Google Workspace or Cloud Identity organization. It can contain folders and directly managed projects.
Folders: Containers that organize projects by department, environment, or policy boundary.
Projects: Primary ownership containers for resources. They hold IAM policies, enabled service APIs, quotas, service usage, and an association with a billing account.
Resources: Individual service resources such as Compute Engine instances, Cloud Storage buckets, Pub/Sub topics, or BigQuery datasets.
Projects can be attached directly to an organization, placed in folders, or initially have no organization parent. A billing account is separate from this hierarchy and can be associated with projects in different organizations.
IAM grants are inherited and additive. A grant at an ancestor level is inherited by descendants and cannot be removed lower down; only an IAM deny policy can block it, and a narrower grant does not override a broader one. Organization policies separately constrain which configurations descendant resources may use.
Resource location affects availability and latency:
Zonal resources operate in one zone.
Regional resources span multiple zones within a region.
Cloud Storage buckets use regional, dual-region, or multi-region locations; a bucket is not itself a global resource.
Global external Application Load Balancers provide a global frontend backed by regional infrastructure.
IAM, Cloud KMS, Secret Manager, Security Command Center, Cloud IDS, VPC Service Controls
IAM, KMS, Secrets Manager, GuardDuty
Entra ID, Key Vault, Defender for Cloud
AI and data science
Vertex AI, Gemini models, Vertex AI Workbench, Data Catalog, Looker
SageMaker, Bedrock, SageMaker Notebooks
Azure AI Foundry, Azure Machine Learning
Choosing Compute and Data Services
Compute Engine: Choose for full control over the guest OS, machine image, networking, disks, and operating-system configuration. It fits legacy applications, specialized runtimes, and workloads that cannot use managed platforms.
GKE: Choose when the team needs Kubernetes APIs, scheduling control, custom operators, or Kubernetes portability. Google manages the GKE control plane, while the worker nodes run on Compute Engine.
Cloud Run: Choose for stateless containers that should scale automatically without operating a cluster. It supports services and jobs and can scale a service to zero when no requests are pending and the minimum instance count is zero.
App Engine: Choose for applications built around its supported runtime and deployment model, particularly managed web services.
Cloud Functions: Choose for small event-driven handlers triggered by Pub/Sub, Cloud Storage, and other supported events. Cloud Run is generally more flexible for larger containers or complex execution requirements.
For databases, Cloud SQL fits conventional relational applications, AlloyDB targets PostgreSQL-oriented workloads, Spanner provides globally distributed strongly consistent transactions, and Firestore or Bigtable suit NoSQL access patterns. BigQuery is a serverless analytics warehouse, not a row-oriented transactional database.
Production Operations and Failure Modes
Google operates the physical infrastructure and managed-service control planes, but customers remain responsible for application configuration, IAM, data handling, secrets, schemas, and workload logic.
Key production practices include:
Run important workloads across multiple zones and, where justified, regions. Use health checks, regional load balancing, and tested failover rather than assuming a zone will remain available.
Design Pub/Sub consumers for at-least-once delivery. Exactly-once delivery is available only for pull subscriptions within a region, so consumers should still be idempotent and use bounded retries and dead-letter handling where appropriate.
Configure database backups, point-in-time recovery, deletion protection where supported, and recurring restore tests. A backup that has never been restored is not a proven recovery strategy.
Use Cloud Monitoring dashboards and SLO-based alerts for availability, latency, saturation, and error-budget consumption. Correlate metrics with Cloud Logging and distributed traces.
Inventory resources with commands such as gcloud projects list and gcloud compute regions list. Keep configuration and dependencies in source control and apply changes through reviewed infrastructure-as-code pipelines.
Plan upgrades deliberately. Managed services reduce patching work, but customer-managed runtime images, client libraries, schemas, and deployment configurations still require updates.
Security and Cost Management
Use separate projects for production, staging, and development. Apply least-privilege IAM roles to groups and service accounts, avoid using the default Compute Engine service account as an application identity, and prefer Workload Identity Federation over downloadable service-account keys. Store sensitive values in Secret Manager.
Use private addresses, restrictive firewall rules, Private Google Access, and VPC Service Controls where the exfiltration risk justifies them. Cloud Armor can protect global external load-balancing endpoints with DDoS protection, web application firewall rules, and rate limiting.
Cost controls include budgets, billing alerts, labels, committed-use discounts, and reviews of idle resources. Budgets are alerts, not hard spending caps. Also monitor network egress, BigQuery bytes processed, storage classes, logging volume, and unused snapshots because these frequently cause unexpected cloud costs.
2
How is the GCP resource hierarchy organized?
KnownRevise
The GCP resource hierarchy organizes an organization’s projects and service resources into a parent-child governance structure. It centralizes IAM and organization policy while retaining projects as the usual boundary for billing, quotas, API enablement, and service configuration.
Read full answerHide full answer
Applies toGoogle Cloud Platform (all current versions)
Hierarchy Structure
The primary hierarchy is:
Organization
The root container for a Google Cloud organization.
Represented as organizations/123456789.
Usually associated with a Google Workspace or Cloud Identity enterprise.
Folders
Group projects and other folders.
Can be nested to represent business units, environments, or governance domains.
Represented as folders/123456789.
Cannot directly contain service resources.
Projects
Required containers for most Google Cloud resources.
Represented as projects/example-project.
Have an immutable project number and a globally unique project ID.
Are the main billing-account association, IAM, API-enablement, quota, and service-configuration boundary.
Resources
Service-specific objects owned by a project, such as Compute Engine instances, GKE clusters, Cloud Storage buckets, Cloud SQL instances, Pub/Sub topics, and BigQuery datasets.
Can have global, regional, or zonal scope.
Do not all implement the same child-resource hierarchy.
A project belongs to exactly one folder or directly to the organization. It cannot belong to multiple folders simultaneously. Organizations and folders are hierarchy nodes; IAM and organization policies attach to supported nodes or resources.
IAM and Organization Policies
Higher-level policies are inherited through the hierarchy.
Policy type
Purpose
Example
IAM allow policy
Grants permissions on supported resources
Allow a group to read project logs
IAM deny policy
Denies permissions regardless of applicable allow policies
Prevent service accounts from deleting production resources
Organization policy
Restricts permitted configurations
Restrict Cloud Storage locations or require uniform bucket-level access
Service Usage constraint
Limits which services can be enabled in descendant projects
Prevent unapproved APIs from being enabled
Important distinctions:
Principals such as users, groups, service accounts, and workload identities are not hierarchy nodes. They receive access through IAM bindings.
Permissions granted at organization, folder, or project level are inherited by supported descendant resources. A lower-level policy cannot subtract a permission granted by an ancestor.
IAM allow policies can be attached to organizations, folders, projects, and resource types that support resource-level IAM. IAM deny policies can be attached at organization, folder, or project level.
IAM permissions and conditional bindings are evaluated when a request is made. Organization-policy constraints are generally enforced during resource creation or update; list constraints can also restrict operations involving existing resources.
Organization policies constrain configuration but do not grant access.
Cloud Resource Manager manages the organization, folder, and project hierarchy. The IAM service manages IAM allow and deny policies, while individual service APIs create and operate resources.
Operational and Security Boundaries
Projects provide useful production boundaries:
Billing: Projects are associated with billing accounts, allowing charges to be consolidated while retaining project-level cost attribution.
Least privilege: Grant permissions at the narrowest practical folder, project, or resource level, while accounting for inherited access.
Environment isolation: Separate development, staging, and production projects, usually beneath separate folders.
Auditability: Cloud Audit Logs, Security Command Center, and Cloud Asset Inventory can operate across the hierarchy.
Data protection: VPC Service Controls and organization policies can protect data across projects.
Lifecycle: Deleting a project affects its resources, but restoration is available only during a limited post-deletion period. Inventory dependencies, export required logs and configuration, and coordinate project retirement before deletion.
The hierarchy does not automatically pool every service quota. Quotas remain service- and scope-specific, so production capacity must be checked for the relevant project, region, and service.
Comparison with AWS and Azure
GCP
AWS
Azure
Organization
Organization
Tenant-level management scope
Folder
Organizational unit
Management group
Project
Account
Subscription
Resources
Service resources with global, regional, or zonal scope
Resources in resource groups, with global or regional scope
All three platforms support nested administrative grouping, but their boundaries differ. AWS organizational units group accounts, while GCP folders group projects and folders. Azure management groups group subscriptions, while Azure resource groups live inside subscriptions and group resources rather than subscriptions.
Production Best Practices
Use the organization for enterprise-wide governance, folders for business units or environments, and projects for independently managed systems.
Apply organization-wide restrictions such as approved regions, uniform bucket-level access, and restricted service APIs.
Grant folder-level permissions only when every descendant project needs the same access; otherwise prefer project or resource scope.
Keep production, non-production, and security-tooling projects in separate folders to simplify policy auditing.
Inventory hierarchy changes with Cloud Asset Inventory and monitor administrative and policy changes through Cloud Audit Logs.
Review inherited IAM and organization-policy behavior during incidents because changes at a parent can affect many descendants.
Treat hierarchy as a governance mechanism, not a backup strategy; separately configure backups, replication, retention, and recovery testing for stateful services.
Check service-specific quotas before placing additional workloads in a project, and plan dependency cleanup before deleting projects or folders.
3
What are regions and zones in Google Cloud?
KnownRevise
A region is a geographic area containing multiple isolated zones, while a zone is an independent deployment and fault domain within a region. Choose a zone for zonal resources and a region for services that replicate or operate across multiple zones.
Read full answerHide full answer
Applies toGoogle Cloud Platform (all current versions)
Regions and Zones
Region: A geographic location identified by a code such as us-central1. It provides a defined area for placing resources while keeping traffic within a predictable geography.
Zone: A deployment area within a region, such as us-central1-a. A region typically contains three or more zones.
Fault isolation: Zones are engineered to be isolated in power, cooling, and network infrastructure. A failure affecting one zone should not directly affect another zone in the same region.
Independent placement: Zone identifiers are region-specific. The -a, -b, and -c suffixes do not imply the same physical location across different regions.
Google Cloud’s logical resource hierarchy—organization, folders, and projects—is separate from geography. A project does not become fault-isolated merely because its name or placement policy refers to a region.
Global external Application Load Balancer frontend
Regional resources do not all provide identical replication, failover, or SLA guarantees. Check the documentation for the specific service rather than assuming that “regional” means fully active-active.
Production Failure Behavior
For a highly available stateless application:
Deploy instances across at least two zones in a region.
Place them behind a suitable load balancer.
Keep regional state in a regional or multi-region service.
Monitor errors and latency separately for each zone.
Deploying multiple instances in one zone does not protect against a zonal outage. Conversely, placing a regional service’s database in only one zone can make the entire application unavailable despite multiple stateless frontends.
For example, a regional Cloud SQL instance with high availability maintains a standby in another zone and manages failover. A GKE regional cluster can distribute nodes across zones, while a zonal cluster depends on one zone. A regional outage requires a multi-region architecture, with additional concerns such as data replication, traffic routing, consistency, and cost.
Regions and Zones Compared with AWS and Azure
Google Cloud
AWS
Azure
Meaning
Region
Region
Region
Geographic area
Zone
Availability Zone
Availability Zone
Isolated deployment area within a region
us-central1
us-east-1
eastus
Region identifier
us-central1-a
us-east-1a
eastus-1
Zone identifier
These mappings are conceptual, not one-to-one. AWS and Azure availability-zone numbering has service- and region-specific interpretations, so infrastructure should never assume equivalent physical failure domains.
Operational Best Practices
Select by requirements: Prioritize user latency, service availability, data residency, compliance, quota, capacity, and regional pricing.
Prefer explicit placement: Include zone or region in infrastructure-as-code, deployment scripts, and incident runbooks.
Avoid shared dependencies: Check whether DNS, certificates, databases, secrets, or internal APIs remain tied to one zone.
Plan upgrades: Stagger maintenance and use service-supported rolling upgrades so one zone is not disrupted simultaneously.
Monitor by location: Alert on zonal error rates, saturation, network failures, and regional SLO burn rather than only aggregate metrics.
Test failover and restoration: Exercise regional database failover, backup restoration, and multi-region traffic routing before an incident.
Control costs: Multi-region replication, cross-region traffic, backups, and regional load balancing increase usage and operational complexity.
4
What is Google Cloud IAM and how are permissions granted?
KnownRevise
Google Cloud IAM is Google Cloud’s authorization system: it grants identities access to resources by binding roles that contain permissions. Authentication establishes who the caller is; IAM determines what that identity may do, while organization policies and resource-specific policies provide additional controls.
Read full answerHide full answer
Applies toGoogle Cloud IAM (all versions)
How Authorization Works
When a principal requests an action on a resource, IAM evaluates the applicable policies:
The request identifies the principal, resource, and requested action.
IAM checks for an allow binding that gives the principal a role containing the required permission.
Any IAM Condition attached to that binding must also be satisfied.
IAM evaluates applicable deny rules. A matching deny rule blocks access even when another policy allows it.
If at least one applicable allow binding grants the permission and no deny rule blocks it, access is granted.
If no policy grants the permission, access is denied by default.
Permissions are granted by adding a binding between a principal and a role at the appropriate scope. Allow policies are additive, so permissions inherited from multiple scopes can combine. Some resources, such as Cloud Storage buckets and Cloud Run services, also expose resource-level IAM controls.
Principals, Roles, and Permissions
Principal: The identity making the request. This can be a Google Workspace user, Google group, service account, workload identity principal, domain, allUsers, or allAuthenticatedUsers.
Permission: A specific operation, such as compute.instances.start or storage.objects.get.
Role: A collection of permissions assigned to a principal through a policy binding.
Binding: Connects a principal to a role and can optionally include an IAM Condition.
Role type
Description
Example
Basic
Grants broad access across most services in the scope where granted
roles/viewer, roles/editor
Predefined
Grants permissions for a specific service or job
roles/compute.viewer, roles/pubsub.subscriber
Custom
Contains permissions selected by an administrator
A role allowing instance creation but not deletion
Avoid roles/owner and roles/editor for routine access because they include broad permissions. Basic roles are generally granted at organization, folder, or project level and should be replaced by narrowly scoped predefined or custom roles where practical.
Projects can also be attached directly to an organization. IAM allow policies are inherited downward: a role granted at the organization level applies to descendant folders, projects, and supported resources; a project-level grant applies to that project’s supported resources.
Deny policies have different inheritance behavior from allow policies. A deny policy is not automatically inherited by child folders, projects, or resources. Attach deny policies at each hierarchy level where denial is required, and verify the intended scope before applying them.
Organization Policy Service rules are separate from IAM. IAM determines who can act, while organization policies determine which configurations and locations are permitted.
A principal and role are bound at an IAM-supported scope
AWS
Identity and resource policies contain permissions
Identity policies attach to principals or roles; resource policies attach to resources
Azure
Role definitions contain actions
Role assignments connect a principal, role, and scope
All three provide deny-by-default authorization, scoped role assignments, hierarchical controls, and audit logging, but their policy models and inheritance rules differ.
Production Best Practices and Failure Modes
Apply least privilege: Prefer narrowly scoped predefined roles, then create custom roles when no suitable predefined role exists.
Use groups: Bind groups instead of individual users so access can be changed centrally without editing many policies.
Use IAM Conditions: Restrict permissions by time, resource attributes, or request properties, including a condition that requires access to expire.
Avoid service-account keys: Use Workload Identity Federation or attached service accounts where supported. Long-lived keys can leak and are difficult to rotate safely.
Separate duties: Give operators, developers, auditors, and incident responders different roles. Use controlled, time-bound elevation for privileged tasks.
Monitor changes: Admin Activity Audit Logs record IAM policy changes. Enable relevant Data Access Audit Logs where resource access must be investigated.
Test changes before removal: Project-level bindings can immediately break CI/CD systems, automation accounts, and production workloads.
Plan for emergency access: Maintain a tightly controlled break-glass process with alerting, short-lived credentials where possible, and regular testing.
Review permissions continuously: Use IAM Recommender to identify unused or excessive permissions, then validate recommendations against workload and disaster-recovery requirements.
5
How do basic, predefined, and custom IAM roles differ in GCP?
KnownRevise
IAM roles are bundles of permissions, but GCP’s basic, predefined, and custom roles differ in scope, precision, and maintenance. Basic roles provide broad access, predefined roles provide Google-maintained job-specific access, and custom roles provide customer-defined permissions.
Read full answerHide full answer
Applies toGoogle Cloud IAM (current stable)
Core Differences
Type
Examples
Granularity
Maintained by
Typical use
Basic roles
roles/viewer, roles/editor, roles/owner
Very broad
Google
Broad access where accepted; controlled break-glass administration
Permissions not covered by a suitable predefined role
Basic roles:
Viewer generally reads resources across many Google Cloud products.
Editor generally creates and changes resources but has less administrative authority than Owner.
Owner includes broad administrative and IAM-management permissions.
They can be granted at the organization, folder, or project level, but not directly to an individual resource.
They expose unrelated data and workloads and are generally unsuitable for routine job access.
Predefined roles:
Google creates and maintains them as products evolve.
They represent common responsibilities, such as deploying a Cloud Run service or administering Compute Engine instances.
Google can update their permission lists, so administrators should monitor changes affecting sensitive permissions.
They are normally the preferred choice for production access.
Custom roles:
They contain only the permissions explicitly selected by the customer.
A custom role can contain at most 280 permissions and can be created at the organization or project level; folder-level custom roles cannot be created.
An organization-level custom role can be granted throughout that organization and on supported descendant resources. A project-level custom role can be granted on its project and supported project resources.
Only permissions supported for use in custom roles can be included. Custom roles are independent definitions, not subsets of basic roles.
Google does not add future permissions automatically, so administrators must review and update the role when services introduce required capabilities.
Resource Hierarchy and Effective Access
IAM evaluates role bindings across the Google Cloud resource hierarchy:
A binding on an organization can apply to every descendant folder, project, and supported resource.
A binding on a folder applies to its descendant folders, projects, and supported resources.
A binding on a project applies to the project and its IAM-supporting resources.
A resource-level binding applies only to that resource.
Allow policies are additive: a member receives the union of permissions granted by applicable bindings. A child project cannot subtract an inherited allow by replacing it with a narrower allow policy. An IAM Deny policy at an applicable organization, folder, or project scope can override allowed actions when that permission and resource support Deny. Organization policies and service-specific controls can impose additional restrictions.
A role therefore does not guarantee that an operation will succeed. VPC Service Controls, organization policies, resource policies, API enablement, and service-side authorization may still block access.
Comparison with AWS and Azure
Capability
Google Cloud
AWS
Azure
Broad predefined access
Basic roles
Broad managed policies; no exact one-to-one role class
Broad built-in roles
Job-specific predefined access
Predefined roles
AWS-managed and customer-managed policies
Built-in roles
Customer-defined permissions
Custom roles
Customer-managed policies
Custom roles
Who maintains the custom definition
Your organization
Your organization
Your organization
The concepts are similar, but they are not identical. An AWS customer-managed policy or Azure custom role is not automatically equivalent to a GCP custom role because scope handling, permission behavior, and administrative safeguards differ.
Production Best Practices
Prefer predefined roles: select the narrowest predefined role that supports the workload, then use IAM Conditions or a custom role when necessary.
Avoid basic roles: especially avoid roles/owner; reserve it for narrowly controlled break-glass administration.
Grant to groups or service accounts, rather than individual users, and keep workload identities separate from human identities.
Review effective permissions, not just direct bindings, because inherited grants can expand access unexpectedly.
Monitor Cloud Audit Logs for IAM policy changes, impersonation, service-account key creation, and access-token generation.
Use IAM Recommender and Security Command Center to identify unused permissions, excessive access, and role assignments that should be reduced.
Test custom-role changes in a non-production project. Updates affect every principal bound to the role after IAM propagation completes.
Maintain an approval and review process for IAM changes, and keep emergency access outside normal CI/CD automation.
6
What are GCP service accounts, and how should they be used?
KnownRevise
A Google Cloud service account is a non-human IAM identity used by workloads and automation. Treat it as a narrowly scoped workload identity: prefer short-lived attached credentials or federation, grant only required permissions, and avoid long-lived keys whenever possible.
Read full answerHide full answer
Applies toGoogle Cloud IAM (current) and Google Cloud CLI (current stable release)
Identity and Authentication
A service account belongs to a project and appears as an IAM principal, for example service-account-name@project-id.iam.gserviceaccount.com. IAM policies bind principals to predefined or custom roles; IAM Conditions can narrow a grant by attributes, time, or resource. It is not a human login, although an authorized principal can impersonate it.
Choose an authentication method based on where the workload runs:
Attached identity: Compute Engine, GKE, Cloud Run, App Engine, and Cloud Functions use the platform’s identity integration to obtain short-lived OAuth 2.0 access tokens without a key file.
Workload Identity Federation: GitHub Actions and other external workloads exchange a configured external credential for a short-lived Google Cloud token.
Impersonation: A principal with roles/iam.serviceAccountTokenCreator can obtain an access token for a service account.
Service account key: A JSON key contains a private key capable of signing authentication assertions. It is a long-lived secret that can be copied or misused, so use it only when attached identities and federation are unavailable.
For every API request, Google Cloud evaluates the token identity, requested operation, resource, IAM policy, applicable IAM Conditions, organization policies, and other controls such as VPC Service Controls. A valid token can still receive 403 Forbidden when authorization is denied.
IAM Roles and Least Privilege
Grant a dedicated identity to each workload or environment rather than sharing platform default accounts.
Need
Suitable role
Read one Secret Manager secret
roles/secretmanager.secretAccessor
Read BigQuery data
roles/bigquery.dataViewer; add roles/bigquery.jobUser if it creates query jobs
Invoke one Cloud Run service
roles/run.invoker
Attach or act as a service account
roles/iam.serviceAccountUser
Mint tokens for a service account
roles/iam.serviceAccountTokenCreator
Use a service account through federation
roles/iam.workloadIdentityUser
Prefer bindings on the specific bucket, dataset, secret, or service over project-wide bindings. Never give routine workloads roles/owner or roles/editor. A binding such as api@example-project.iam.gserviceaccount.com plus roles/storage.objectViewer grants only the permissions in that role, and conditions can further constrain it.
Comparison with AWS and Azure
Concept
Google Cloud
AWS
Azure
Workload identity
Service account
IAM role
Managed identity or service principal
Runtime attached identity
Service account attached to a VM, GKE workload, or Cloud Run service
IAM role for EC2, EKS, or Lambda
System-assigned or user-assigned managed identity
External workload access
Workload Identity Federation
IAM OIDC or SAML identity provider with AssumeRoleWithWebIdentity
This example creates a dedicated Cloud Run identity, grants access to one Secret Manager secret, and attaches the identity to the service.
export PROJECT_ID="example-project"
export REGION="us-central1"
export SERVICE_NAME="example-api"
export SA_NAME="example-api-production"
export SECRET_NAME="example-api-secret"
export IMAGE="us-docker.pkg.dev/example-project/example-services/example-api:latest"
gcloud config set project "${PROJECT_ID}"
gcloud iam service-accounts create "${SA_NAME}" \
--display-name="Example API production identity"
export SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
# Generate a demo secret; provision the real application credential securely.
openssl rand -base64 32 | \
gcloud secrets create "${SECRET_NAME}" \
--replication-policy=automatic \
--data-file=-
# Grant access only to this secret.
gcloud secrets add-iam-policy-binding "${SECRET_NAME}" \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/secretmanager.secretAccessor"
gcloud run deploy "${SERVICE_NAME}" \
--image="${IMAGE}" \
--region="${REGION}" \
--service-account="${SA_EMAIL}" \
--set-secrets="APP_CREDENTIAL=${SECRET_NAME}:latest" \
--no-allow-unauthenticated \
--quiet
The deploying principal needs Cloud Run deployment permissions, permission to act as the service account, and access to the secret version being mounted. Cloud Run captures standard container stdout and stderr; a logging-writer role is not needed merely for that output.
Production Operations and Failure Modes
Use dedicated identities: Separate production, staging, deployment, and telemetry identities to improve attribution and limit blast radius.
Treat keys as exceptions: If unavoidable, store keys in a secrets manager, restrict access, rotate them, and revoke them promptly. Prefer attached identities or federation.
Monitor audit activity: Review Admin Activity and enabled Data Access audit logs. Alert on service-account changes, key creation, token minting, and repeated authorization failures.
Diagnose failures: 401 usually indicates missing, invalid, expired, or revoked credentials; 403 indicates authorization or policy denial. Disabled or deleted identities and failed federation conditions also cause immediate failures.
Plan for policy propagation: IAM and organization-policy changes can take several minutes to propagate, so avoid create-attach-delete workflows that require immediate effect.
Protect metadata access: Treat the metadata server as a privileged interface. Prevent SSRF vulnerabilities from allowing untrusted workloads to request arbitrary metadata-server paths.
7
What is Workload Identity Federation in Google Cloud?
KnownRevise
Workload Identity Federation (WIF) lets external workloads authenticate to Google Cloud by exchanging credentials issued by an external identity provider for short-lived Google Cloud access tokens. It removes the need to distribute or store Google Cloud service account keys in CI/CD systems, hybrid environments, Kubernetes clusters, and other workloads outside Google Cloud.
Read full answerHide full answer
Applies toGoogle Cloud IAM Workload Identity Federation (all current versions)
How It Works
An external identity provider authenticates the workload and issues a signed, short-lived identity token or assertion.
The workload presents that credential to Google Cloud’s Security Token Service for the configured workload identity pool and provider.
Google Cloud validates the credential’s signature, issuer, audience, expiry, and any mapped attributes.
The provider’s attribute mapping converts provider-specific claims into standard attributes such as google.subject. Its attribute condition can restrict which claims are accepted.
Google Cloud evaluates the resulting principal against IAM policy bindings.
The workload receives a short-lived access token, either by impersonating a service account or by using a federated principal directly on an IAM-enabled resource.
The trust relationship is therefore based on verifiable identity and contextual claims rather than a copied long-lived secret.
Core Components
Workload identity pool: A Google Cloud resource representing a trust boundary for external identities. Pools can be global or regional.
Provider: Defines how Google Cloud validates an external issuer, including its audience, attribute mappings, and attribute condition. WIF supports OIDC, SAML 2.0, and AWS identity sources.
Attribute mapping: Converts claims such as assertion.sub or a repository claim into attributes used in principal identifiers and conditions.
Attribute condition: A CEL expression that narrows acceptable identities, for example to one repository, AWS role, issuer, or audience.
Service account impersonation: The common pattern grants roles/iam.workloadIdentityUser on a service account to a matching federated principal. The workload can then request an access token for that service account.
Direct resource access: A federated principal can receive permissions directly on resources such as Cloud Storage buckets or BigQuery datasets without impersonating a service account.
Workforce Identity Federation: A separate Google Cloud mechanism for federating human users, such as employees, and is not the subject of WIF.
Comparison with AWS and Azure
Platform
Federation mechanism
Typical identity source
Credential outcome
Google Cloud WIF
Pool, provider, attribute mapping, condition, and IAM bindings
OIDC, SAML 2.0, AWS, GitHub, Entra ID
Service account impersonation token or direct resource access token
AWS IAM OIDC
OIDC identity provider plus IAM role
Kubernetes, GitHub, other OIDC providers
Temporary AWS role session
AWS IAM Roles Anywhere
Trust anchor, profile, and trust policy
X.509 or AWS STS web identity
Temporary AWS role session
Microsoft Entra Workload ID Federation
Federated identity credential attached to a managed identity
External OIDC issuer
Microsoft Entra token for the managed identity
GCP WIF is structurally similar to AWS IAM OIDC because both exchange an external token for temporary cloud credentials. GCP uses pool and provider resources to define the trust relationship, while AWS represents it with an IAM identity provider and role trust policy. Azure attaches a federated credential to a managed identity.
Production Operations
Monitoring: Review available Cloud Audit Logs entries for IAM and token-exchange activity, together with the external provider’s authentication logs.
Failure diagnosis: Common causes include an incorrect issuer or audience, missing claims, an attribute condition that no longer matches, clock skew, an invalid principal identifier, or missing IAM bindings.
Token lifetime: External credentials and exchanged Google Cloud tokens should be short-lived. Service account impersonation is preferable to distributing static keys.
Revocation: Removing the service account IAM binding prevents further impersonation by that principal, although it does not necessarily prevent the underlying federated-token exchange. Deleting the provider prevents new exchanges using that provider.
Blast-radius control: Use separate pools or providers for unrelated trust domains and environments so a provider or condition change does not affect every consumer.
Incident response: Assume that already issued access tokens may remain usable until expiry. Disable the service account when necessary, revoke access through IAM policy changes, and use short token lifetimes to limit exposure.
Infrastructure as code: Manage pools, providers, conditions, service accounts, and IAM bindings through reviewed Terraform or equivalent automation.
Best Practices and Pitfalls
Require an exact issuer and explicit allowed audiences rather than trusting claims from a broad issuer.
Use narrow attribute conditions based on stable identifiers such as repository, AWS role, or workload identity.
Grant roles/iam.workloadIdentityUser only on the service account the principal must impersonate.
Give the impersonated service account only the permissions required by the workload; avoid roles such as roles/owner.
Prefer principal sets that match specific attributes over broad wildcard identities.
Test new repositories, roles, issuers, audiences, and conditions in a separate provider or pool before production rollout.
Never place identity tokens, credential configurations, or service account keys in logs or source control.
8
How do IAM conditions and organization policies work in Google Cloud?
KnownRevise
IAM conditions make an IAM role binding conditional, while organization policies enforce guardrails on permitted operations and resource configurations. Both can restrict access, but conditions determine whether a particular request qualifies, whereas organization policies determine whether an action or configuration is allowed within a hierarchy scope.
Read full answerHide full answer
Applies toGoogle Cloud IAM and Organization Policy Service (current stable); Google Cloud CLI (current stable)
Core Difference
IAM condition: Adds a Common Expression Language (CEL) expression to a role binding. The principal receives the binding’s permissions only when that expression evaluates to true.
Organization policy: Defines a constraint at an organization, folder, or project. It can restrict IAM permissions, disable risky capabilities, or enforce configuration rules.
Combined evaluation: IAM must allow an operation. If it does, an applicable organization policy can still prevent the operation or configuration.
Neither grants access by itself: IAM grants permissions; organization policies can only restrict what IAM permits.
IAM Conditions
IAM conditions are useful when the same role should have different effects depending on the resource, request, or time.
For example, an operations team might receive roles/compute.instanceAdmin.v1, but only for Compute Engine instances whose resource names begin with projects/example-project/zones/us-central1-a/instances/prod-. The condition could evaluate resource.type == "compute.googleapis.com/Instance" together with resource.name.startsWith("projects/example-project/zones/us-central1-a/instances/prod-").
Key CEL attributes include:
resource.type: The resource type being accessed.
resource.name: The resource’s full resource name.
request.time: The time of the API request.
request.auth.claims: Identity-provider claims when supplied and exposed by the relevant service.
Attribute availability depends on the target resource and service. A condition referring to an unsupported attribute cannot be used for that resource.
A condition affects only the binding that contains it. If the same principal has an unconditional binding for the same role at the organization, folder, project, or resource level, that broader grant can still provide access. Basic roles such as roles/owner, roles/editor, and roles/viewer do not support IAM conditions.
Organization Policies
Organization policies provide hierarchy-wide guardrails. Attach them at the organization, folder, or project level; applicable constraints are inherited by child folders and projects and their resources. The policies affect requests involving those resources rather than being inherited by principals as hierarchy objects.
Typical uses include:
Preventing user-managed service account key creation.
Restricting which identity domains may appear in IAM policy bindings.
Preventing public exposure of storage resources or public IP configuration for databases.
Restricting resource locations, external IP addresses, or other configurable properties.
For example, disabling service account key creation prevents the key-creation operation even for a principal whose IAM role would normally allow it. Google Cloud recommends Workload Identity Federation, service account impersonation, or other short-lived credential mechanisms instead of downloadable user-managed service account keys.
Organization policies do not grant permissions. Use them for platform-wide controls and IAM for identity-specific authorization. A lower-level configuration cannot simply relax a stricter ancestor constraint, so inspect the effective inherited policy.
Comparison With AWS and Azure
Concern
Google Cloud
AWS equivalent
Azure equivalent
Conditional role grants
IAM Conditions with CEL
IAM condition keys and policy conditions
Azure RBAC conditions and ABAC
Hierarchy-level permission limits
Organization policy constraints
Service Control Policies
Azure Policy and management-group assignments
Resource configuration controls
Organization policy constraints, including custom constraints
AWS Config rules and control mechanisms
Azure Policy
Main distinction
IAM permits identities; organization policies restrict operations and configuration
SCPs primarily cap IAM permissions
Azure Policy primarily governs resource configuration and selected permissions
Production Best Practices and Failure Modes
Test conditions first: Validate CEL expressions against representative resources. A condition that matches resource names can be broader than expected if principals can create similarly named resources.
Check all applicable grants: Search organization, folder, project, and resource-level bindings. A conditional project grant does not override an unconditional inherited grant.
Roll out guardrails gradually: Test organization policies in a nonproduction folder or controlled project before enforcing them across critical workloads.
Preserve emergency access: Maintain tested break-glass identities and ensure their access remains compatible with each policy.
Audit changes: Use Cloud Audit Logs to detect IAM and organization policy modifications, and alert on unexpected changes.
Review effective permissions: IAM Recommender and Policy Intelligence can help identify overbroad roles and assess guardrail impact.
Back up IAM policies: Export policies before bulk changes so rollback does not depend on reconstructing them manually.
Use least privilege: Prefer groups over individual users, narrowly scoped roles, short-lived credentials, and Workload Identity Federation over long-lived user-managed service account keys.
9
How do you apply least privilege in Google Cloud?
KnownRevise
Least privilege in Google Cloud means granting each identity only the permissions it needs, on the narrowest practical resource, and only for as long as necessary. Implement it with narrowly scoped IAM roles, groups, service identities, IAM Conditions, and deny policies.
Read full answerHide full answer
Applies toGoogle Cloud IAM and gcloud CLI (current stable releases)
How Least Privilege Works
Google Cloud IAM combines who is requesting access, the role containing allowed permissions, and the resource on which the role is granted.
Resource hierarchy: Organizations contain folders, folders contain projects, and projects contain resources such as Cloud Run services, buckets, and Cloud SQL instances.
Inheritance: Roles granted at an organization, folder, or project level are inherited by its descendants. A broad parent-level grant can therefore expose many resources.
Narrow scope: Grant a role at the lowest resource where it is needed and where that role supports resource-level binding.
Time and attribute constraints: IAM Conditions can restrict access by time, resource attributes, request attributes, or identity attributes.
Explicit denial: IAM Deny policies override allow policies, providing defense in depth. They should supplement—not replace—narrowly scoped grants.
Separation of duties: Platform administrators, security administrators, developers, and workload identities should not share one highly privileged account.
Avoid roles/owner, roles/editor, and roles/viewer for routine access. These basic roles cover broad sets of GCP services and make privilege review and incident containment difficult.
Selecting Roles and Identities
Use predefined roles when they closely match the job requirement. Create custom roles only when predefined roles contain unnecessary permissions.
Task
Prefer
Appropriate scope
Deploy Cloud Run services
roles/run.developer
Specific service or project
Invoke Cloud Run services
roles/run.invoker
Specific service
Read Cloud Storage objects
roles/storage.objectViewer
Bucket or managed folder
Connect to Cloud SQL
roles/cloudsql.client
Project, narrowed with an IAM Condition on the instance name
Read BigQuery datasets
roles/bigquery.dataViewer
Dataset or table
Access a secret
roles/secretmanager.secretAccessor
Specific secret
View logs
roles/logging.viewer
Lowest supported scope
Identity practices are equally important:
Put human users in groups rather than granting individual user accounts.
Create a dedicated service account for each workload or environment.
Use Workload Identity Federation or workload identity mechanisms instead of downloadable service account keys.
Grant roles/iam.serviceAccountTokenCreator only to identities that must impersonate a service account.
Use separate deployment, runtime, and data-access service accounts.
Prefer IAM Conditions for temporary elevation, maintenance windows, or access limited to particular resources.
GCP, AWS, and Azure Equivalents
GCP
AWS
Azure
IAM role binding
IAM policy containing role and principal
Azure RBAC role assignment
Service account
IAM role assumed through a trust policy
Managed identity or service principal
IAM Condition
IAM policy condition
Azure role assignment condition
Resource hierarchy
Organization, account, resource hierarchy
Management group, subscription, resource group
The concepts are similar, but role names, condition attributes, inheritance behavior, and administrative APIs remain provider-specific.
Code Example
The following grants deployment permission only on one Cloud Run service:
gcloud config set project example-project
gcloud run services add-iam-policy-binding order-service \
--region=us-central1 \
--member="group:release-engineers@example.com" \
--role="roles/run.developer"
# Allow a workload identity to invoke only this service.
gcloud run services add-iam-policy-binding order-service \
--region=us-central1 \
--member="serviceAccount:orders@example-project.iam.gserviceaccount.com" \
--role="roles/run.invoker"
# Inspect recent IAM audit events.
gcloud logging read \
'resource.type="project" AND protoPayload.serviceName="iam.googleapis.com"' \
--project=example-project \
--limit=20
If predefined roles are too broad, a project-level custom role can contain only the required permissions:
gcloud iam roles create orderDeployer \
--project=example-project \
--title="Order deployer" \
--description="Deploy and inspect Cloud Run services" \
--permissions=run.services.create,run.services.get,run.services.update \
--stage=GA
Production Operations and Best Practices
Manage policies declaratively: Store IAM bindings in Terraform or another authoritative policy source. Detect and remove manual configuration drift.
Audit continuously: Enable relevant Cloud Audit Logs, export security-critical logs to a protected destination, and alert on policy changes, principal additions, and unusual privileged actions.
Review regularly: Re-evaluate permissions after role changes, service migrations, upgrades, and employee or contractor offboarding.
Test failure modes: Overly restrictive policies commonly cause HTTP 403 responses, Cloud Run invocation failures, database connection errors, or startup failures. Check audit logs and the effective inherited policy before broadening access.
Handle policy changes carefully: IAM updates can have a brief propagation delay. Avoid retry loops that repeatedly change policy or accidentally replace existing bindings.
Plan emergency access: Use controlled, monitored break-glass accounts rather than permanent broad grants. Time-bound IAM Conditions can reduce the exposure window.
Protect organization administration: Keep organization-level administration separate from day-to-day project access, and restrict billing and cost-management permissions so workloads cannot change billing configuration.
Review custom roles: Custom roles require explicit maintenance when APIs or required permissions change. Prefer predefined roles when their privilege level remains appropriate.
A strong least-privilege design is not simply the smallest possible role set; it is the smallest stable, auditable, and operationally reliable permission set that supports the workload.
10
What does Google Cloud Compute Engine provide?
KnownRevise
Google Cloud Compute Engine is Google Cloud’s Infrastructure as a Service (IaaS) platform for running virtual machines with configurable CPU, memory, storage, networking, and operating systems. It provides the control of a traditional server while offloading facilities, hypervisors, and much of the hardware management to Google Cloud.
Read full answerHide full answer
Applies toGoogle Cloud Compute Engine (current GA service)
Core Concepts
VM instances: Individual virtual machines that run Linux or Windows workloads. Each instance belongs to a specific Google Cloud zone, such as us-central1-a.
Machine types: Predefined or custom combinations of vCPU and memory. Burstable machine types can share physical cores and provide lower cost, while memory-optimized and compute-optimized types suit databases, analytics, and batch processing.
Storage: Instances can use Persistent Disk, Hyperdisk, or ephemeral local storage. Persistent disks outlive VM deletion when configured appropriately.
Images and snapshots: Boot from a Google-provided image, a custom image, or an existing disk. Snapshots provide point-in-time recovery and can be used to create images or new disks.
Networking: Compute Engine attaches instances to a VPC network through regional subnets. Instances can have internal and external IP addresses, firewall rules, Cloud NAT, and load-balancer backends.
Automation: Managed instance groups deploy identical VMs, perform health checks, replace unhealthy instances, and support rolling or canary upgrades.
Resource and Security Model
Google Cloud resources normally follow this hierarchy:
Organization → Folders → Projects
Compute Engine resources are created in a project. VPC networks are global, subnets are regional, and VM instances are zonal. Shared VPC lets one host project own shared networking while service projects contain their workloads.
Access is controlled through IAM and VPC firewall rules:
IAM controls who can administer or attach instances to projects.
Firewall rules control network traffic between instances and the internet.
Service accounts give applications a Google Cloud identity with narrowly scoped permissions.
OS Login manages SSH authentication through IAM rather than relying only on instance metadata.
Shielded VM provides secure boot, virtual trusted platform modules, and integrity monitoring where supported.
Avoid using the default service account for production applications. Instead, create a dedicated service account and grant only the permissions the workload needs.
High Availability and Production Operations
A single VM instance is a single point of failure. Common deployment patterns include:
Regional managed instance group: Spreads instances across multiple zones in one region and can automatically replace unhealthy VMs.
Global load balancing: Routes traffic to healthy backends in multiple regions for internet-facing HTTP(S) applications.
Multi-region deployment: Runs independent regional deployments for larger regional-failure coverage. A regional managed instance group does not survive the loss of its entire region.
Use Cloud Monitoring for instance metrics, uptime checks, dashboards, and alerting, and Cloud Logging for diagnostics. Alert on service-level indicators such as request errors, latency, and saturation rather than relying only on CPU utilization.
For recovery, schedule Persistent Disk snapshots and store them regionally or multi-regionally when required. Snapshots are block-level backups: quiesce databases or coordinate application-level backups when consistent recovery points matter.
Failure Modes and Trade-offs
Zone outage: Instances in that zone may become unavailable. Regional instance groups reduce this risk within a region.
Bad deployment: An image or startup script can make every newly created instance unhealthy. Use canary deployments and validate images before broad rollout.
Capacity or quota exhaustion: A zone may lack machine types, or the project may hit quotas for CPUs, disks, addresses, or load-balancer resources. Monitor quota usage and reserve capacity where justified.
Networking lockout: Incorrect firewall rules or IAM changes can block administration. Maintain controlled break-glass access.
Storage failure: Use appropriate disk types and replication, monitor disk health, and test restoration from snapshots.
Cost growth: Idle VMs, provisioned disks, snapshots, egress, and load balancers continue generating charges. Apply labels, budgets, and anomaly detection.
Maintenance: Eligible instances support live migration, but kernel updates, license changes, and some specialized configurations may still require restart or replacement. Managed rolling updates simplify fleet maintenance.
Comparison With AWS and Azure
Need
Google Cloud
AWS
Azure
General-purpose VMs
Compute Engine
EC2
Azure Virtual Machines
Kubernetes
GKE
EKS
AKS
Serverless containers
Cloud Run
ECS/Fargate or App Runner
Azure Container Apps
Fully managed application platform
App Engine
Elastic Beanstalk
App Service
Choose Compute Engine when the workload needs operating-system control, custom networking, specialized machine shapes, persistent VMs, or self-managed software. Prefer GKE for Kubernetes orchestration, Cloud Run for serverless containers, and App Engine when the platform should own most deployment and scaling decisions.
Best Practices
Deploy across multiple zones for production availability.
Use managed instance groups and rolling updates rather than managing every VM manually.
Apply least-privilege IAM, dedicated service accounts, and restricted firewall rules.
Enable OS Login or centrally managed SSH credentials.
Automate infrastructure with Terraform or Infrastructure Manager (Deployment Manager is deprecated) and version configurations.
Define monitoring, budgets, snapshot policies, and tested recovery procedures before launch.
Tag resources for ownership, environment, cost allocation, and automated policy enforcement.
11
What are Compute Engine machine families and custom machine types?
KnownRevise
Machine families group Compute Engine machine types optimized for particular workloads, while custom machine types let you select an allowed combination of vCPUs and memory when no predefined configuration fits.
Read full answerHide full answer
Applies toGoogle Cloud Compute Engine (current stable); Google Cloud CLI current stable; Terraform 1.x with the `google` provider 5.x+
Machine Family Concepts
A machine family represents a workload and hardware profile, such as general-purpose, compute-optimized, storage-optimized, or accelerator-optimized.
Compute-heavy workloads requiring more vCPUs relative to memory
z3
High-throughput storage workloads
a3
GPU-accelerated workloads
A machine type is an exact configuration within a family. For example, n2-standard-4 specifies:
Family: n2
Shape: standard, representing a predefined vCPU-to-memory ratio
vCPUs: 4
Memory: The corresponding predefined amount of RAM
The machine type controls CPU and memory, but does not determine boot-disk type, disk throughput, VPC network, or external IP configuration. Those are separate instance settings.
Custom Machine Types
A custom machine type lets you specify:
The number of vCPUs
The amount of memory in MiB
For example, n2-custom-4-8192 requests 4 vCPUs and 8,192 MiB of memory. Custom configurations are not completely arbitrary: the vCPU count and memory amount must satisfy the selected family’s minimum, maximum, and allowed memory-per-vCPU limits.
Custom types are useful when a workload needs, for example, 4 vCPUs with 6 GiB of RAM when the nearest predefined type provides substantially more memory. They do not customize the underlying processor model, physical-core allocation, GPU configuration, or storage.
Choosing Between Options
Option
When to use it
Main trade-off
Predefined machine type
Capacity and ratio already match the workload
Simpler sizing and often predictable cost
Custom machine type
A valid intermediate CPU-memory ratio is required
More configuration work; unused RAM is still billed
Shared-core type
Small, intermittent, or low-utilization services
CPU sharing and throttling behavior
Accelerator-optimized type
The workload requires attached GPUs
Additional cost and capacity constraints
Use a custom type only after checking its effective hourly cost against the closest predefined type. Custom types are regular VM instances and are not shared-core or burstable machine types.
Code Example
Create and inspect a custom N2 instance:
# Create an N2 custom instance with 4 vCPUs and 8 GiB of RAM.
gcloud compute instances create gcp-custom-vm \
--project="example-project-id" \
--zone="us-central1-a" \
--machine-type="n2-custom-4-8192" \
--image-family="ubuntu-2404-lts-amd64" \
--image-project="ubuntu-os-cloud" \
--boot-disk-size="20GB" \
--boot-disk-type="pd-balanced"
# Confirm the configured machine type.
gcloud compute instances describe gcp-custom-vm \
--project="example-project-id" \
--zone="us-central1-a" \
--format="value(machineType)"
Capacity is zonal: a custom configuration can be available in one zone but lack capacity in another. Before deployment, verify both CPU and memory quotas and account for possible zone-level capacity failures.
Use Cloud Monitoring to track sustained CPU utilization, memory pressure, disk throughput and latency, network throughput, and instance-level errors. Autoscaling should use measured application demand rather than vCPU count alone.
Changing an instance’s machine type requires stopping it and running gcloud compute instances set-machine-type; for repeatable production deployments, change the instance template and roll it out through a managed instance group instead. Back up mutable disks with snapshots or image-based backup procedures rather than relying on the VM lifecycle.
Best Practices
Benchmark with representative CPU, memory, and I/O workloads.
Compare custom sizing with predefined alternatives using actual billing data.
Reserve memory for workload growth instead of choosing the smallest possible custom shape.
Check regional quota and capacity before committing to a topology.
Use separate machine types for separate performance domains rather than relying on overcommitted shared instances.
Recreate instances through automation so machine-type upgrades are repeatable and auditable.
12
What are Compute Engine Spot VMs and preemption?
KnownRevise
Preemptible VMs are discounted Compute Engine instances that Google Cloud may terminate to reclaim capacity. Spot VMs are the current Compute Engine name and provisioning model for this behavior, providing low-cost but interruptible compute capacity.
Read full answerHide full answer
Applies toGoogle Cloud Compute Engine Spot VMs; current stable gcloud CLI and Terraform Google provider
How Preemption Works
A Spot VM runs on temporarily available capacity rather than capacity reserved specifically for the instance. Google Cloud may preempt it when that capacity is needed elsewhere.
The lifecycle is:
The VM starts normally using standard machine types, disks, networking, and IAM identities.
Google Cloud sends a preemption notice, normally allowing approximately 30 seconds before termination.
The instance is terminated, and compute billing stops.
Persistent disks are not deleted by the termination itself, but local SSD data is lost.
Treat the notice as best-effort. Applications must not depend on receiving every notification or completing cleanup successfully. “Preemptible” describes the termination behavior; “Spot” is the current product terminology and SPOT provisioning model.
Spot VMs Compared with Standard VMs
Property
Spot VM
Standard VM
Pricing
Deeply discounted
On-demand pricing
Capacity
Uses spare capacity that may be reclaimed
Not preempted for capacity reclamation
Interruption
May receive a 30-second termination notice
No capacity-driven preemption
SLA
No Spot-specific SLA
Eligible for the applicable Compute Engine SLA
Best suited for
Batch jobs, CI workers, testing, rendering
Critical services and predictable workloads
The corresponding services in other major clouds are AWS Spot Instances and Azure Spot VMs. Their economic model is similar: low prices, interruptibility, and no guarantee that capacity can be obtained.
Cloud
Service
Approximate interruption notice
Google Cloud
Compute Engine Spot VMs
30 seconds
AWS
EC2 Spot Instances
About 2 minutes
Azure
Azure Spot VMs
At least 30 seconds (Scheduled Events)
Production Behavior
Use for interruptible work: Stateless APIs, asynchronous queues, CI runners, data processing, development environments, and rendering workloads are typical candidates.
Handle failure explicitly: Make jobs idempotent, checkpoint progress, and retry them on another instance after preemption.
Externalize state: Store durable state in Cloud SQL, Firestore, Bigtable, Cloud Storage, or another managed service instead of relying on the VM or local SSD.
Use instance groups for availability: A Managed Instance Group can replace preempted Spot VMs and combine Spot and Standard provisioning for graceful degradation.
Expect capacity failures: A Spot request can remain unscheduled even when standard capacity and project quota are available.
Monitor interruptions: Alert on preemption and termination events, Spot capacity failures, queue latency, and unsuccessful job retries.
Protect data: Configure disk retention and backups correctly, and use customer-managed encryption keys when required by policy.
Code Example
Create a standalone Spot VM with the current gcloud provisioning model:
Do not use Spot VMs as the only copy of important data. Configure persistent-disk retention, automated backups, and tested restoration procedures.
Design for abrupt termination. Graceful shutdown hooks are useful but cannot guarantee that cleanup completes.
Apply least privilege. Attach a dedicated service account with only the permissions the workload requires, and avoid broad access to production secrets.
Patch and scan images: Spot status does not reduce the need for vulnerability management, trusted images, and timely Google Cloud security updates.
Control costs: Set budgets and alerts, and monitor effective compute cost per completed job rather than only the hourly instance price.
Maintain fallback capacity: Use mixed or Standard-based instance groups when workload continuity is more important than minimizing compute cost.
13
What are managed instance groups in Google Cloud?
KnownRevise
Managed instance groups (MIGs) are Compute Engine features that manage a collection of interchangeable virtual machines from an instance template. Google Cloud automatically creates, deletes, scales, health-checks, and replaces those instances according to the group’s configuration.
Instance template: Defines the VM configuration, including machine type, boot image, network, service account, metadata, labels, firewall tags, and startup script.
Managed instance group: Controls the number and lifecycle of instances created from the template.
Autoscaler: Optionally changes the target size between configured minimum and maximum limits.
Health check: Determines whether instances are healthy and, when autohealing is enabled, causes unhealthy instances to be replaced.
Rolling update: Gradually replaces instances after a new instance template is applied.
The group manages instances individually, but they should behave as interchangeable members. A MIG does not itself distribute traffic; attach it as the backend of a load balancer or use it with a service that targets Compute Engine instances.
Zonal and Regional MIGs
Type
Scope
Availability characteristics
Typical use
Zonal MIG
One zone
Can lose all instances in that zone
Cost-sensitive or single-zone workloads
Regional MIG
Multiple zones in one region
Spreads instances across zones in the region
Production services requiring zonal resilience
A regional MIG still operates in one Google Cloud region. Multi-region resilience requires separate regional MIGs, usually placed behind a global external Application Load Balancer.
Scaling, Health, and Deployments
Fixed-size groups: Maintain a specified number of instances and replace failed instances when the group detects problems.
Autoscaled groups: Adjust capacity using signals such as CPU utilization, Cloud Monitoring metrics, or a schedule.
Autohealing: A failed or unhealthy instance is deleted and recreated from the template. A MIG health check alone marks instances unhealthy; it does not necessarily replace them unless autohealing is configured.
Rolling updates: New instances are launched using an updated template before old instances are removed, subject to capacity and rollout policies.
Changing an instance template does not modify existing instances by itself. The updated template must first be applied to the managed instance group.
Service Comparison and Selection
Google Cloud
AWS
Azure
Important distinction
Managed instance group
Auto Scaling group
Virtual Machine Scale Set
GKE node pools use MIGs internally
GKE node pool
EKS managed node group
AKS node pool
Choose when Kubernetes APIs and workload scheduling are required
Cloud Run
ECS/Fargate
Azure Container Apps
Choose for serverless containers without VM management
Cloud Functions
Lambda
Azure Functions
Choose for event-driven functions
Use a Compute Engine MIG when the team needs direct control over VMs, guest operating systems, networking, disks, or specialized instance configuration.
gcloud CLI Example
The following creates a service account, instance template, zonal MIG, autoscaler, and health check. The referenced subnet must already exist.
# Create a dedicated identity for the VM instances
gcloud iam service-accounts create example-mig
# Define the desired instance configuration
gcloud compute instance-templates create web-v1 \
--machine-type=e2-standard-2 \
--image-family=ubuntu-2404-lts-amd64 \
--image-project=ubuntu-os-cloud \
--region=us-central1 \
--subnet=example-subnet \
--tags=http-server,https-server \
--service-account=example-mig@example-project.iam.gserviceaccount.com
# Create a managed group with two initial instances
gcloud compute instance-groups managed create web-mig \
--zone=us-central1-a \
--template=web-v1 \
--size=2 \
--base-instance-name=web
# Allow automatic scaling between two and ten instances
gcloud compute instance-groups managed set-autoscaling web-mig \
--zone=us-central1-a \
--min-num-replicas=2 \
--max-num-replicas=10 \
--target-cpu-utilization=0.6
# Configure an application-level health check and autohealing
gcloud compute health-checks create http web-health \
--port=80 \
--request-path=/healthz
gcloud compute instance-groups managed update web-mig \
--zone=us-central1-a \
--health-check=web-health \
--initial-delay=300
After creating an updated template such as web-v2, apply it with a rolling update:
Use regional MIGs for important services, and separate regional deployments for protection from a regional outage.
Keep instances stateless: replacement destroys local disks and local application state. Use Cloud SQL, Spanner, Firestore, Bigtable, or Filestore where durable state is required.
Make startup scripts idempotent: a replacement instance may rerun them and must recover cleanly.
Monitor capacity and health: alert on compute.googleapis.com/instance_group/managed/instance_count, autoscaler behavior, backend health, CPU, memory, disk pressure, and load-balancer latency.
Account for quota and scale-out delay: autoscaling cannot immediately bypass regional Compute Engine quotas or provisioning capacity limits.
Control rollout risk: use canary groups or weighted backend services when a change requires gradual validation.
Apply least privilege: use a dedicated service account, avoid instance-wide SSH keys and embedded credentials, retrieve secrets from Secret Manager, and restrict access with IAM and VPC firewall rules.
Plan upgrades deliberately: publish a new image or template, validate it, and use a rolling update rather than relying on in-place guest patching.
14
How does Compute Engine autoscaling work?
KnownRevise
Managed instance group (MIG) autoscaling adjusts the number of Compute Engine instances in a managed instance group based on workload metrics. A Managed Autoscaler monitors Cloud Monitoring data, then creates or removes instances using the group’s instance template while keeping capacity within configured minimum and maximum limits.
A MIG manages interchangeable instances created from an instance template.
An autoscaling policy defines the target metric, such as average CPU utilization or a custom Cloud Monitoring metric.
The autoscaler periodically reads aggregated metrics for the group.
If utilization exceeds the target, it increases the group’s desired size, subject to stabilization and cooldown rules.
If utilization remains below the scale-in threshold, it decreases the desired size when the stabilization policy permits.
New instances use the instance template, while removed instances are terminated and can be replaced if the group needs to return to its desired size.
The autoscaler manages MIG capacity rather than scaling standalone virtual machines. A regional MIG distributes capacity across zones and targets a configured distribution; a zonal MIG operates within one zone.
Scaling Policies and Cloud Equivalents
Capability
Compute Engine
AWS
Azure
Managed workload group
Regional or zonal MIG
Auto Scaling group
Virtual Machine Scale Set
Capacity definition
Instance template
Launch template or configuration
VM image and configuration
Scaling signal
CPU, Cloud Monitoring metrics, and other supported signals
Target tracking, schedules, and predictive scaling
Autoscale rules and schedules
Multi-zone placement
Regional MIG
Auto Scaling group spanning Availability Zones
VMSS with zone spreading
Initial capacity
target-size
Desired capacity
Default capacity
Common policies include:
CPU utilization target: Scales according to average CPU usage across the group. A lower target such as 50% reserves more headroom and normally triggers scale-out sooner than a 70% target; a higher target tolerates more utilization per instance but may react later.
Custom metric: Scales on a scalar application metric exposed to Cloud Monitoring, such as requests per second, queue depth, or Pub/Sub backlog.
Minimum size: Maintains baseline capacity for expected traffic and reduces cold-start latency.
Maximum size: Limits potential compute cost and runaway scaling, although quotas and regional capacity can impose additional limits.
Cooldown period: Prevents another scale-up for a defined period after a scale-out. The default is 60 seconds.
Production Behavior and Failure Modes
Cold starts matter: Autoscaling creates instances, but they become useful only after the guest OS, application startup, and readiness checks complete. Capacity may exist before it can serve traffic.
Scale-in can remove active instances: Keep sessions, jobs, and durable state outside the VM unless the application handles instance termination safely.
Health checks are separate from scaling: A MIG can replace unhealthy instances according to its health-checking configuration, while the autoscaler changes capacity according to load.
Thrashing is possible: An unrealistic metric target or long application startup time can cause repeated scale-out and scale-in cycles.
Regional quotas can block scale-out: Increasing max-size does not guarantee that Compute Engine can obtain the required CPU, memory, or IP addresses.
Rolling updates and scaling are separate: Changing an instance template affects future instances but does not automatically modify existing MIG instances. To adopt the change, update the MIG instances through a rolling-update or instance-update operation, validate application compatibility, and configure connection draining.
Monitor continuously: Alert on sustained maximum capacity, repeated instance recreation, failed health checks, excessive scale-out activity, and slow application readiness. Also track latency and error rates, not only CPU.
Security, Cost, and Best Practices
Put the application behind a regional load balancer that uses the MIG as a backend and checks application readiness.
Set min-size according to steady-state traffic and latency requirements. Every idle instance continues consuming compute and storage charges.
Use a dedicated service account with least-privileged IAM roles for the application. Do not place secret values in templates, startup scripts, or instance metadata.
Ensure Google-managed service agents retain the service-specific permissions required by the autoscaling and instance-group workflows.
Use custom metrics when CPU does not represent demand. Prefer a suitable scalar metric such as request rate or Pub/Sub backlog rather than relying only on CPU.
Test scale-out under realistic startup conditions and verify autoscaling behavior against quotas, regional capacity, and downstream dependency limits.
Prefer an autoscaler for variable production workloads, but retain capacity planning for predictable peaks, maintenance windows, and large launches.
15
What is a Google Compute Engine instance template?
KnownRevise
A Compute Engine instance template is a reusable, regional definition of the configuration used to create Compute Engine virtual machines. It specifies settings such as the machine type, boot disk, network, identity, metadata, and scheduling options, but it does not run a VM itself.
Read full answerHide full answer
Applies toGoogle Compute Engine API v1 and Google Cloud CLI (current stable)
How It Works
An instance template is a named regional resource with a URL such as projects/PROJECT_ID/regions/REGION/instanceTemplates/TEMPLATE_NAME. A single-instance creation can reference it through sourceInstanceTemplate, while a Managed Instance Group (MIG) version can reference its regional template URL. Compute Engine resolves that reference during provisioning and creates the instance using the stored configuration.
Reusable configuration: Multiple instances or MIGs can use the same template.
Regional scope: The template belongs to a region, while the instance or MIG selects an eligible zone within that region.
Persistent resource: The template remains available until explicitly deleted.
No automatic deployment: Creating or updating a template does not modify existing instances; instances adopt it only when newly created or replaced.
No direct runtime cost: The template is control-plane configuration, although the instances and other resources it provisions are billable.
Versioning templates by name, such as web-v1 and web-v2, is a common operational convention rather than a separate template-versioning API.
What a Template Defines
Area
Common settings
Compute
Machine type, CPU platform, scheduling, and Shielded VM configuration
Storage
Boot disk, source image or snapshot, disk type, disk size, and additional disks
Networking
Network, subnetwork, additional interfaces, and network tags
Identity
Service account and OAuth scopes
Software
Image selection, metadata, and startup scripts
Automation
Labels, resource-manager tags, and settings shared by provisioned instances
An instance template is more than an OS image. An image supplies the boot filesystem, while the template supplies the complete VM definition. Pinning a specific image or a controlled custom image gives stronger reproducibility; using an image family can make provisioning depend on the current image selected for that family.
Service Comparisons
GCP concept
Closest equivalent
Main distinction
Compute Engine instance template
AWS EC2 Launch Template
Both define reusable VM configuration consumed by instance groups or auto-scaling groups
Compute Engine instance template
Azure VM scale set model
Azure normally incorporates image and VM configuration into the scale-set model rather than a standalone template resource
Cloud Run revision
AWS Lambda function version, Azure Container Apps revision
Intended for managed container serving rather than general-purpose VMs
GKE node pool
EKS node group, AKS node pool
Configures Kubernetes workers rather than standalone Compute Engine instances
Production Behavior and Failure Modes
Existing instances remain unchanged: A MIG template change does not itself perform a rollout; use a controlled rolling replacement or recreate instances.
Bad templates can repeat failures: Autohealing may repeatedly recreate unhealthy instances from the same defective definition.
Boot success is not readiness: A startup script can complete while its application is failing to start. Monitor application-level readiness separately.
Health checks need accurate targets: An incorrect port or path can cause unnecessary instance replacement and an incident.
Capacity and quotas can block rollouts: Replacement surges can exceed regional or zonal capacity and Compute Engine quotas.
Images and policies can expire: Deprecated images, organization policies, or policy denials can prevent new instances from being created.
Regional MIGs need spare capacity: Leaving headroom in multiple zones supports maintenance events and zone-level disruption.
Durable data belongs outside instances: Use persistent disks with scheduled snapshots for disk-backed workloads, or an appropriate managed database for critical state.
Production teams generally maintain canary and stable templates, observe rollout health, and retain the previous template until the replacement is proven. Templates should be managed through reviewed infrastructure-as-code, using Terraform’s google_compute_instance_template resource or Config Connector desired-state resources reconciled by Config Controller.
Security and Operational Practices
Use a dedicated service account with least-privilege IAM roles, and avoid downloadable service-account keys.
Limit OAuth scopes to those the workload actually requires.
Use OS Login and Identity-Aware Proxy TCP forwarding for administrative access rather than broad internet-facing SSH rules.
Never embed credentials in startup scripts, metadata, or templates; retrieve secrets through an approved secret-management path.
Treat startup scripts and custom images as privileged code because they run with administrative privileges and can alter the host.
Install the Ops Agent through the image or configuration pipeline so metrics and logs remain consistent across instances.
Configure the MIG health check against a meaningful readiness signal; keep application liveness detection separate so transient failures do not automatically trigger replacement.
Apply organization policies, retention controls, and change review to templates and the images they reference.
16
What does Google Kubernetes Engine (GKE) provide?
KnownRevise
Google Kubernetes Engine (GKE) is Google Cloud’s managed Kubernetes service. Google operates the Kubernetes control plane, integrates clusters with Google Cloud IAM and networking, and provides options for automatically managing or self-managing worker nodes so teams can run containerized production workloads.
Read full answerHide full answer
Applies toGoogle Kubernetes Engine (GKE), all currently supported versions
How It Works
GKE runs standard Kubernetes components, including the API server, scheduler, controller manager, and etcd. A regional cluster distributes the control plane across multiple Google Cloud zones, while a zonal cluster places it in one zone and is less expensive but has a smaller failure domain.
Worker nodes run as Compute Engine instances. With VPC-native alias IP ranges, Pods receive addresses from dedicated VPC subnets without relying on traditional Google Cloud instance routes. Kubernetes Services can expose applications through Google Cloud load balancers, while internal traffic remains inside the VPC.
Typical operational commands include:
gcloud container clusters describe CLUSTER_NAME --region REGION
gcloud container clusters get-credentials CLUSTER_NAME --region REGION
gcloud container node-pools list --cluster CLUSTER_NAME --region REGION
Deployment Options
Mode
Node management
Billing
Best suited for
GKE Autopilot
Google manages node pools, versions, scaling, and node configuration
Primarily based on requested Pod resources, with additional charges for some resources
Production workloads where teams want less cluster administration
GKE Standard
Customers configure node pools, machine types, and upgrade or network settings; GKE manages the node image lifecycle
Cluster fees plus Compute Engine, storage, and network costs
Environments requiring custom node configurations, specialized hardware, or node-level control
GKE On-Prem
Runs in an on-premises environment and is managed through Google Cloud tooling
Separate GKE On-Prem pricing model
Regulated, edge, or disconnected environments requiring Kubernetes with Google Cloud integration
Autopilot still permits standard Kubernetes manifests, Deployments, Services, and custom resources. It constrains node access and configuration because Google manages the node layer.
Core Platform Capabilities
Horizontal and vertical scaling: HPA adjusts replicas from observed demand; VPA recommends or applies resource-request changes.
Cluster autoscaling: Adds or removes nodes when Pods cannot be scheduled, subject to quota, capacity, and autoscaling-policy constraints.
Release channels: Rapid, Regular, Stable, and Extended channels provide different upgrade cadences. Extended supports longer-lived release lines.
Multi-cluster management: Fleet management organizes clusters and supports standardized configuration and policy enforcement.
Ingress and routing: GKE Ingress and the GKE Gateway controller integrate with Google Cloud load balancing.
Storage integration: Persistent disks and Cloud Storage integrations support stateful workloads, but application teams remain responsible for data protection.
Network policy: Kubernetes NetworkPolicy and Google Cloud firewall rules provide separate layers of network control.
Production Operations
GKE upgrades the control plane first and then the node pools, because nodes may not run a newer version than the control plane. In a zonal cluster the single control-plane replica is briefly unavailable during its upgrade; a regional cluster upgrades its replicas one at a time, so the Kubernetes API stays available.
In GKE Standard, teams should use maintenance windows, node-upgrade settings, and Pod disruption budgets rather than relying only on manual node replacement. Autopilot handles more of the lifecycle but does not eliminate the need to test application compatibility.
Production clusters should include:
Resource requests and limits: Required for scheduling, bin-packing, and capacity planning.
Health checks: Configure startup, readiness, and liveness probes so traffic reaches only healthy Pods.
Pod disruption budgets: Protect critical replicas during voluntary disruptions such as node maintenance.
Monitoring and logging: Use Cloud Monitoring and Cloud Logging, with alerts for saturation, crashes, rollout failures, and unavailable replicas.
Backup strategy: Kubernetes does not automatically back up application data. Use database-native backups, volume snapshots, or application-level replication according to recovery objectives.
GKE combines Kubernetes RBAC with Google Cloud IAM. Standard Workload Identity for GKE maps a Kubernetes service account to a narrowly scoped IAM service account that the Pod can impersonate without downloadable service-account keys. Workload Identity Federation for GKE supports external identities through federation and can optionally use service-account impersonation.
Additional controls include:
Private clusters: Restrict control-plane access to authorized networks and a private endpoint.
Shielded GKE Nodes: Apply hardened node protections such as secure boot and integrity monitoring.
Least privilege: Grant only required IAM and Kubernetes permissions; avoid broad roles such as Owner or Editor.
Audit logging: Enable relevant Kubernetes and Google Cloud audit logs and monitor changes to RBAC, workloads, secrets, and network policy.
Secret management: Use Secret Manager instead of placing long-lived credentials in images or manifests.
Policy enforcement: GKE Policy Controller can detect insecure configurations before deployment.
GKE, EKS, and AKS
Platform
Cloud
Main advantage
Main trade-off
GKE
Google Cloud
Integrated networking, IAM, load balancing, and operations
Google-specific integrations can increase portability concerns
EKS
AWS
Strong AWS service and ecosystem integration
Teams commonly manage more node and add-on infrastructure themselves
AKS
Azure
Strong Microsoft Entra ID and Azure integration
Azure-specific identity and networking features can require platform-specific design
Choose GKE when workloads already use Google Cloud services such as BigQuery, Cloud SQL, Pub/Sub, or Cloud Storage, or when its managed node operations and networking reduce operational overhead. For a multi-cloud platform, standardize on Kubernetes manifests and isolate provider-specific APIs to preserve portability.
17
How do GKE Standard and GKE Autopilot differ?
KnownRevise
GKE Standard gives teams direct control over Compute Engine node pools, machine types, and capacity, while GKE Autopilot manages the worker infrastructure using pod resource requests. Both run standard Kubernetes workloads on a Google-managed control plane; the primary difference is who manages the worker nodes and how capacity is provisioned and billed.
Read full answerHide full answer
Applies toGoogle Kubernetes Engine (GKE) Standard and Autopilot, current supported GKE releases
Core Difference
GKE Standard: Teams create and manage node pools backed by Compute Engine instances. Node-pool size, autoscaling, machine types, labels, taints, disk configuration, and upgrade behavior are directly configurable.
GKE Autopilot: Google manages the complete worker-node lifecycle, including provisioning, configuration, bin-packing, and upgrades. Teams deploy Kubernetes resources and specify CPU, memory, and ephemeral-storage requests.
Shared behavior: Both expose Kubernetes APIs and support workloads such as Deployment, StatefulSet, and DaemonSet, with a Google-managed control plane.
Important distinction: Autopilot is not serverless Kubernetes. Applications still use Kubernetes manifests, APIs, RBAC, networking, and workload controllers.
Feature Comparison
Area
GKE Standard
GKE Autopilot
Worker nodes
Customer-managed Compute Engine instances
Google-managed and hidden from direct administration
Capacity planning
Node count, node-pool shape, and machine type
Pod resource requests and an optional compute class
Autoscaling
Cluster autoscaler and optional node auto-provisioning
Automatic capacity based on scheduled workloads
Billing
Provisioned node resources, including unused capacity
Requested CPU, memory, and ephemeral storage, subject to per-pod minimums
Node customization
Broad control over images, labels, taints, disks, and hardware
Opinionated configuration with workload restrictions
Upgrades
Control plane managed by Google; node upgrade timing configurable
Control plane and worker lifecycle handled by Google
Operational effort
Higher
Lower
Best fit
Precise infrastructure or capacity control
Conventional services seeking less node-management work
How Autopilot Works
Autopilot evaluates scheduled pods and groups them onto Google-provisioned workers. Capacity is added and removed as pod resource requests change.
Resource requests drive placement and cost: If CPU, memory, or ephemeral-storage requests are omitted, Autopilot applies defaults of 0.1 vCPU, 128 MiB memory, and 1.5 GiB ephemeral storage. Explicit, realistic requests are still important for scheduling and cost control.
Compute classes select capacity profiles: Workloads can use platform-defined general-purpose, balanced, memory-optimized, or accelerator-optimized classes, subject to regional and quota availability.
Node-level access is restricted: Privileged containers, host networking, hostPort, and host-path mounts are not supported. GKE sandbox can provide stronger workload isolation, but it does not expose unsupported node interfaces.
Workloads must remain node-agnostic: Applications should not depend on node names, manually installed host agents, arbitrary custom node labels, or a particular machine type.
Production Trade-offs
Standard advantages:
Precise control over machine types, local disks, custom node images, GPUs or TPUs, and node agents.
Dedicated node pools with taints, affinity, and labels for specialized workloads.
Reserved or preallocated capacity when predictable steady-state capacity is required.
Standard failure modes:
Node exhaustion, uneven rollout distribution, and autoscaler latency.
Configuration drift and stale images across node pools.
Disk pressure from logs, images, or poorly managed ephemeral storage.
Higher cost when allocated nodes remain underused.
Autopilot advantages:
Google handles node provisioning, maintenance, and replacement.
Lower operational effort and less directly provisioned idle capacity.
Faster onboarding for teams already familiar with Kubernetes.
Autopilot failure modes:
Overstated requests increase cost, while understated requests cause CPU throttling or OOM kills, because Autopilot sets limits equal to requests by default.
Sudden traffic increases can still encounter scale-out latency.
Node-dependent workloads may require redesign.
Requested resources and per-pod minimums remain billable even when utilization is low.
Selection and Operational Guidance
Choose Standard when workloads require specific machine shapes, local storage, custom node configuration, specialized agents, or tight control over capacity and scheduling.
Choose Autopilot for conventional services when reducing node-management overhead matters more than controlling exact node configuration. Autopilot supports selected accelerators, but its hardware and placement choices are more constrained than Standard.
Before migration, audit privileged containers, host networking, host paths, custom node labels, machine-type assumptions, storage drivers, and DaemonSets. DaemonSets are supported, but their resource requests and placement must fit the managed-node model.
Monitor CPU throttling, OOM kills, pending pods, scheduling latency, utilization, and requested-versus-used resources. Store manifests in version control and back up application data; Google manages the platform, not application-level recovery.
18
What is Google Cloud Run and when should you use it?
KnownRevise
Cloud Run is Google Cloud’s serverless container platform. It runs container images behind managed HTTPS endpoints, automatically scales instances, and can scale to zero when idle. Google operates routing, certificates, and compute; teams own the application, image, configuration, and data dependencies.
Read full answerHide full answer
Applies toGoogle Cloud Run (current stable service; no numbered releases) and Cloud Run functions (2nd gen)
Core Capabilities
Cloud Run services: Managed deployments of stateless containers that serve HTTP requests through managed ingress or receive events through Eventarc.
Cloud Run functions (2nd gen): Event-driven handlers built on Cloud Run and Eventarc, invoked by Pub/Sub, Cloud Storage, and other supported event sources.
Cloud Run jobs: Batch or administrative workloads that run to completion, with parallel tasks where appropriate; unlike services, they do not serve requests.
Revisions: Each deployment produces an immutable revision. Traffic can be split between revisions for canary testing or shifted back for rollback.
Container contract: The container must listen on the port supplied through the PORT environment variable and bind to 0.0.0.0. Logs written to stdout and stderr reach Cloud Logging.
When to Choose Cloud Run
Requirement
Recommended option
Stateless HTTP API or microservice
Cloud Run
Pub/Sub, Eventarc, or webhook processing
Cloud Run functions (2nd gen) for new projects
Containerized background task
Cloud Run Jobs
Predictable low latency with warm capacity
GKE or Compute Engine
Persistent disks or local state
Compute Engine or GKE with persistent storage
Privileged workloads or extensive cluster control
GKE
Simple source-based PaaS deployment
App Engine
Cloud Run is closely comparable to AWS App Runner and Azure Container Apps. AWS Lambda and Azure Functions are function-first alternatives, while GKE provides more operating-system and networking control at the cost of cluster administration.
Production Behavior
Autoscaling: Instances scale from zero to meet demand. Maximum-instance limits protect downstream systems and constrain resource consumption.
Cold starts: Image startup, runtime initialization, and application initialization add latency. A minimum of one instance keeps capacity warm but incurs idle cost.
Concurrency: Multiple requests can share an instance. Higher concurrency improves utilization but increases memory pressure and contention.
Resource allocation: By default, CPU is allocated during request processing, while memory remains allocated for the instance’s lifetime. Cloud Run Jobs allocate CPU throughout execution.
Timeouts: Service request timeouts can be configured up to 60 minutes. Long-running requests consume capacity and can trigger downstream timeouts.
Observability: Cloud Logging captures standard output and errors; Cloud Monitoring exposes service metrics for rate, errors, latency, and instance count; Cloud Trace and Error Reporting assist diagnosis.
Security and Failure Modes
IAM: Grant roles/run.invoker only to identities, groups, or service accounts that require invocation rather than exposing the service publicly.
Ingress: Use internal-only or controlled ingress when the API should not be directly reachable from the internet.
Service identity: Run each service with a dedicated, least-privilege service account rather than the broad default compute identity.
Secrets: Store credentials in Secret Manager and inject them at runtime; never bake them into container images.
Network controls: Use Direct VPC egress when the service requires controlled access to internal resources.
Supply chain: Scan images in CI and deploy by digest, such as image@sha256:..., because version tags can be overwritten.
Failure handling: Design handlers to tolerate cold starts, memory exhaustion, excessive concurrency, dependency failures, and unhandled termination signals. Make side effects idempotent because clients and event delivery may repeat work.
State and recovery: Container filesystems are ephemeral. Store durable state in Cloud SQL, Firestore, Spanner, Bigtable, or Cloud Storage and assume instances can disappear.
Deployment and Operations
A private deployment can use gcloud run deploy api --image=us-central1-docker.pkg.dev/example-project/apps/api@{{IMAGE_DIGEST}} --region=us-central1 --service-account=api@example-project.iam.gserviceaccount.com --no-allow-unauthenticated --ingress=internal-and-cloud-load-balancing --min-instances=1 --concurrency=80.
After testing a revision in CI, shift production traffic with gcloud run services update-traffic api --to-revisions {{REVISION_NAME}}=100 --region=us-central1.
For slow initialization, enable startup CPU boost where appropriate and design the application to become ready reliably. Handle SIGTERM gracefully, alert on user-visible symptoms such as p95 latency and error rate, and rehearse revision rollback before an incident requires it.
Best Practices and Cost Controls
Set explicit CPU, memory, concurrency, timeout, and maximum-instance limits.
Keep images small, use multi-stage builds, remove unnecessary components, and scan dependencies.
Use separate services and service accounts for separate trust boundaries.
Make handlers idempotent and bound retries with exponential backoff and jitter.
Use Cloud Run Jobs for batch work instead of keeping a service alive solely to process it.
Monitor costs with labels and budgets; budgets alert on spend but do not cap it. Minimum instances, sustained request processing, memory, and outbound networking all affect charges.
Avoid Cloud Run when the workload requires local disks, privileged host access, guaranteed ultra-low latency without warm capacity, or extensive cluster-level control.
19
How does Cloud Run auto-scale, and what causes cold starts?
KnownRevise
Cloud Run uses request-based autoscaling to add or remove container instances as traffic changes. A cold start occurs when Cloud Run must create and initialize a new container instance, adding latency to the first request.
Read full answerHide full answer
Applies toGoogle Cloud Run and Google Cloud CLI (current stable versions)
How Cloud Run Scales
Request routing: Cloud Run sends incoming requests to available instances of the active revision.
Concurrency: Each instance can handle multiple simultaneous requests. The default concurrency is 80, with a configurable maximum of 1000.
Scale-out: When request concurrency exceeds the capacity of the current warm pool, Cloud Run adds instances. Requests can briefly queue while capacity is provisioned, and scale-out stops at the service’s maximum instance limit.
Scale-in: When demand falls, Cloud Run removes unused instances.
Scale-to-zero: By default, a service can scale to zero instances after becoming idle. The next request triggers a cold start.
Minimum instances: Setting min-instances to a value greater than zero keeps that many instances warm, reducing cold starts but creating idle cost.
Revision independence: During a gradual rollout, each revision can receive different traffic and may require its own warm capacity.
The autoscaler is primarily driven by incoming request concurrency and traffic rather than by a traditional target-CPU metric.
What Causes a Cold Start
A cold start can include several stages:
Provisioning: Cloud Run creates a new execution instance.
Image retrieval: The container image is pulled, potentially from Artifact Registry or another registry.
Container initialization: The container runtime starts the process and runs its entry point.
Application initialization: The application loads configuration, establishes database connections, initializes SDK clients, and warms caches.
Readiness: Once the container is ready, Cloud Run routes requests to it.
A large container image, slow registry access, heavy framework initialization, or unavailable downstream services can make initialization significantly slower.
Cold starts can occur both:
When a service scales from zero after inactivity.
When traffic causes the autoscaler to add instances beyond the already-warm pool.
min-instances=1 prevents scale-to-zero cold starts for that revision, but it does not prevent cold starts on newly added instances during a sudden burst.
Key Trade-offs
Setting
Benefit
Risk
Low concurrency
Lower per-instance memory and latency risk
More instances, higher cost, more scale-out activity
High concurrency
Fewer instances for I/O-heavy workloads
Longer requests, CPU contention, or out-of-memory failures
min-instances=0
Lowest idle cost
Cold-start latency
min-instances > 0
Predictable latency and faster startup
Continuous idle charges
Low max-instances
Strong cost control
Capacity exhaustion and possible HTTP 429 responses
High max-instances
Better burst absorption
Potentially very high cost if traffic or retries are uncontrolled
Cloud Run uses request-based CPU allocation by default, meaning CPU is generally allocated only while processing requests. Applications that perform continuous background work should evaluate instance-based CPU allocation instead.
Comparison with AWS and Azure
Platform
Primary model
Scale-to-zero
Typical use case
Google Cloud Run
Managed containers
Supported
Stateless APIs, web services, event processing
AWS Lambda
Managed functions
Supported
Short event-driven functions
Azure Functions
Managed functions
Supported
Event-driven and scheduled functions
AWS ECS/Fargate
Containers
Not natively request-based; desired count can be zero
Long-running container workloads
Azure Container Apps
Containers and revisions
Supported
Microservices with event-driven scaling
Choose Cloud Run when the workload is containerized and benefits from serverless request scaling without maintaining a server or cluster. Choose a service such as GKE or AWS ECS when long-running processes, specialized networking, or deep cluster control are more important.
Production Practices
Control scale-out cost: Set an appropriate max-instances value and configure budget alerts. Remember that retries and traffic spikes also create instances.
Avoid accidental public access: Use Cloud Run IAM and grant roles/run.invoker only to trusted identities or service accounts. Add --allow-unauthenticated only for intentionally public services.
Monitor capacity and latency: Track run.googleapis.com/request_count, run.googleapis.com/request_latencies, run.googleapis.com/container_instance_count, CPU utilization, memory utilization, and concurrency. Alert on elevated tail latency, sustained 429 responses, and abnormal instance growth.
Minimize startup work: Use small images, avoid unnecessary initialization, defer nonessential work, and cache dependencies where safe. Keep startup-time network calls to reliable, low-latency dependencies.
Warm new revisions carefully: During traffic splitting, direct a small amount of traffic to the new revision before increasing it so startup failures and latency are detected early.
Keep state external: Treat container filesystems as ephemeral. Store durable state in Cloud SQL, Spanner, Firestore, Bigtable, or Cloud Storage.
Plan for dependency failures: A database or API outage during initialization can cause startup failures, request errors, and repeated cold starts even when Cloud Run itself is healthy.
20
How do Cloud Run, Cloud Functions, and App Engine compare?
KnownRevise
Cloud Run, Cloud Functions, and App Engine are Google Cloud’s main application-compute options, optimized for containers, event handlers, and managed applications respectively. The key distinction is the abstraction each platform exposes: an arbitrary container service, a function with a managed trigger, or an application-oriented PaaS.
Read full answerHide full answer
Applies toGoogle Cloud Run (current stable), Cloud Functions (2nd gen), App Engine Standard and Flexible (current stable)
Service Comparison
Dimension
Cloud Run
Cloud Functions
App Engine
Primary model
Serverless container execution
Event-driven function execution
Managed application platform with Standard and Flexible variants
Invocation
HTTP; Eventarc/CloudEvents; Pub/Sub push and Cloud Scheduler through HTTP or Eventarc
HTTP, Google Cloud events, Pub/Sub, and schedules
HTTP; Standard also supports cron and Task Queues, while Flexible is commonly called through HTTP or Cloud Scheduler
Runtime
A container image supported by Cloud Run, listening on the port supplied through $PORT
Supported language runtimes managed by Google
Standard uses managed language runtimes; Flexible uses Compute Engine VMs and can run custom container runtimes
Scaling
Automatic scaling, including scale-to-zero
Automatic scaling, including scale-to-zero
Standard automatic scaling can reach zero; Flexible uses VM-based automatic scaling and does not scale to zero
Deployment artifact
Container image or source built through Cloud Build
Function source package with its dependencies
app.yaml; Flexible can additionally use a custom runtime container
Best fit
APIs, microservices, custom containers, and event processors
Small event handlers and integration glue code
Existing PaaS applications or teams wanting application-level deployment and scaling management
Execution and Scaling
Cloud Run starts containers on Google-managed infrastructure. If no minimum instances are configured, a service can scale to zero, so the first request may incur a cold start. Revisions are immutable, which makes traffic splitting and rollback straightforward.
Cloud Functions 2nd gen is built on Cloud Run but adds a function-oriented programming model and managed event triggers. Google Cloud invokes a specific function and entry point rather than treating the deployment as an arbitrary HTTP container.
App Engine Standard provides managed language runtimes; its second-generation runtimes use Cloud Run technology while retaining App Engine’s application lifecycle, deployment, and scaling model. App Engine Flexible runs applications in Compute Engine VMs, offering more operating-system freedom but less aggressive scale-to-zero behavior.
Cloud Run and Cloud Functions are regional resources. An App Engine application’s location is selected when the application is created and can affect latency and regulatory placement.
Deployment and Traffic Management
Cloud Run supports immutable revisions, weighted traffic splitting, gradual rollouts, and traffic rollback.
Cloud Functions creates revisioned deployments, but it does not provide Cloud Run-style weighted traffic splitting between function revisions.
App Engine provides application versions and routing through service configuration and dispatch rules.
Cloud Run and Cloud Functions use IAM invoker permissions; public HTTP invocation requires an explicit grant to allUsers. App Engine access controls must be configured separately, and an HTTPS redirect does not itself make an application private.
AWS and Azure Equivalents
Google Cloud service
Closest AWS equivalent
Closest Azure equivalent
Important distinction
Cloud Run
AWS App Runner
Azure Container Apps
Cloud Run also resembles ECS/Fargate but adds managed request-based scaling and scale-to-zero
Cloud Functions
AWS Lambda
Azure Functions
Google Cloud provides tightly integrated event sources and trigger types
App Engine
AWS Elastic Beanstalk
Azure App Service
App Engine is more opinionated and provides an integrated application lifecycle
These mappings are not one-to-one. App Runner is primarily a managed web-service platform, while ECS/Fargate exposes more infrastructure-level control.
Production Guidance
Retries and idempotency: Event delivery can be repeated. Use deduplication keys, transactional updates, or idempotency records, and make retries safe. Returning an error from an event-triggered function commonly causes another delivery attempt.
State and recovery: These compute platforms are not durable database or backup systems. Keep authoritative state in services such as Cloud SQL, Firestore, Spanner, Bigtable, or Cloud Storage, and configure backups, point-in-time recovery, replication, or versioning as appropriate.
Observability: Monitor request latency, error rate, instance count, CPU utilization or throttling, and billing with Cloud Monitoring, Cloud Logging, Error Reporting, and Cloud Trace. Derive cold-start behavior from startup traces or logs rather than assuming a dedicated cold-start metric.
Capacity and cost: Review quotas, concurrency, and maximum-instance limits; exhaustion can appear as 429 responses or increased latency. Cloud Run, Cloud Functions, and App Engine Standard can scale to zero, whereas App Engine Flexible normally retains at least one instance. Minimum-instance settings reduce cold starts but create idle charges.
Security and upgrades: Use least-privilege runtime service accounts, grant invocation rights only to required identities, store secrets in Secret Manager, and restrict internal ingress where appropriate. Google patches managed language runtimes, while teams operating their own containers remain responsible for image maintenance, vulnerability scanning, and runtime upgrades.
21
How do App Engine standard and flexible environments differ?
KnownRevise
App Engine standard is a tightly managed platform for supported language runtimes, while App Engine flexible runs applications in flexible compute instances and supports custom Docker-based runtimes. Standard is usually simpler and cheaper; flexible provides more operating-system, networking, and runtime control.
Read full answerHide full answer
Applies toGoogle Cloud App Engine standard (second-generation runtimes) and flexible environments (current service versions)
Core Differences
Area
Standard environment
Flexible environment
Runtime model
Google-managed runtimes for supported languages
Managed instances with custom containers or supported runtimes
Deployment
Source deployment using app.yaml and the configured runtime
Deployment using app.yaml, commonly with env: flex and runtime: custom
Customization
Restricted by the runtime sandbox
Broader control through Dockerfiles, processes, libraries, and entrypoints
Filesystem
Read-only except writable temporary storage such as /tmp
Writable but ephemeral instance filesystem
Scaling
Predefined instance classes and managed autoscaling
Configurable instance resources, minimum instances, and autoscaling
Workloads
HTTP APIs, web applications, and lightweight services
Applications needing custom runtimes, more control, or longer-running processing
Cost profile
Lower platform overhead and simpler instance pricing
Greater instance overhead, but more control over allocated resources
Runtime and Request Model
Standard environment applications run inside Google-managed language and framework environments. This produces predictable startup and deployment behavior, but applications must comply with runtime restrictions. Unsupported native libraries, system packages, or custom processes usually require choosing a different runtime or flexible environment.
Flexible environment provides a conventional compute environment inside App Engine. Applications can use Docker, install operating-system packages, run multiple processes where supported, and define an entrypoint. The instance filesystem is still ephemeral, so neither environment should be treated as durable storage.
Standard is well suited to REST APIs, admin portals, and request/response applications. Flexible is preferable when the application requires a custom runtime, specialized native dependencies, or more control over the execution environment.
Scaling, Networking, and Deployment
Both environments integrate with App Engine’s global application routing and managed HTTPS. They support versioned deployments, traffic splitting, and rollback:
gcloud app deploy deploys a standard application from its configuration directory.
gcloud app deploy app.yaml deploys a flexible application whose app.yaml contains env: flex; runtime: custom also requires a valid Dockerfile and successful image build.
gcloud app versions list shows deployed versions.
gcloud app traffic set --splits=example-version=1.0 directs all traffic to the selected deployed version.
Autoscaling handles demand changes in both environments, but flexible applications have more explicit control over instance size and baseline capacity. Configure minimum instances only when cold starts are unacceptable and the resulting fixed cost is justified.
Networking differs materially: second-generation standard uses a Serverless VPC Access connector for private egress and does not support direct VPC egress, while flexible supports direct VPC egress. By default a connector carries only private-range traffic; if it is set to route all egress (egress_setting: all-traffic), internet-bound traffic then needs Cloud NAT. Verify private DNS, regional routing, and quotas as well.
Choosing a GCP Service
Requirement
Recommended option
Simple supported runtime with minimal operations
App Engine standard
Custom Docker image with App Engine-managed instances
App Engine flexible
Event-driven HTTP containers or scale-to-zero
Cloud Run
Managed job execution
Cloud Run Jobs
Full Kubernetes orchestration
GKE
Fully managed relational database
Cloud SQL
Managed document database
Firestore
Managed wide-column NoSQL database
Cloud Bigtable
App Engine standard is conceptually closer to platforms such as Azure App Service or traditional Heroku-style PaaS. Flexible is closer to a managed container or VM platform, although it retains App Engine’s deployment and traffic-management model. For a new stateless container where scale-to-zero is important, Cloud Run often provides a better fit than flexible.
Production Operations and Security
Monitoring: Use Cloud Monitoring for request latency, error rate, instance utilization, and autoscaling behavior. Use gcloud app logs tail --service=default for rapid log inspection and Cloud Logging for retention and analysis.
Availability: Deploy a new version and shift traffic gradually rather than replacing all instances at once.
Backups: Application files are not a backup system. Use Cloud SQL automated backups or point-in-time recovery and, for Cloud Storage, versioning with suitable retention or replication; versioning alone is not an independent disaster-recovery copy.
Security: Use least-privilege IAM, a dedicated service account, Secret Manager, and narrowly scoped API permissions. The App Engine service account represents the application, not the signed-in end user.
Upgrades: Runtime deprecations require migration before the published deadline. For flexible, rebuild and test the container whenever the base image, dependencies, or runtime configuration changes.
Migration and Common Pitfalls
Moving from standard to flexible is an architectural migration, not just a configuration change. Validate filesystem assumptions, background processing, startup time, health checks, outbound networking, memory limits, and cost before shifting production traffic.
The most common failures are selecting flexible unnecessarily, assuming local files persist, ignoring runtime or regional quotas, and enabling minimum instances without accounting for their continuous cost. Test the exact deployment artifact, use gradual traffic migration, and retain rollback capacity during production releases.
22
What are Cloud Functions in GCP and what triggers do they support?
KnownRevise
Cloud Run functions (formerly Cloud Functions, 2nd gen) are event-driven, single-purpose compute units. A deployed function runs in response to an HTTP request or a supported Google Cloud/Eventarc event, can scale to zero when idle, and is backed by a Cloud Run service.
Read full answerHide full answer
Applies toCloud Run functions 2nd gen (Cloud Run API v1 and Cloud Functions API v2), current stable Google Cloud
How It Works
Build and runtime: Source is built into a container through Cloud Build and deployed as a Cloud Run service, inheriting Cloud Run’s IAM, revisions, networking, and observability model.
Event delivery: Non-HTTP provider events pass through Eventarc as CloudEvents 1.0. Eventarc filters can select attributes such as type, bucket, and name.
Scaling and limits: Functions scale to zero by default. min-instances reduces cold starts, max-instances limits blast radius, and concurrency defaults to 1 with a maximum of 1000. Supported memory ranges from 128 MiB to 32 GiB; timeout defaults to 60 seconds and can reach 3600 seconds.
Security and networking: Functions use a service account and require invoker authorization. HTTP functions are private by default. Default ingress allows internal and Cloud Load Balancing traffic; private API access can use a VPC connector or direct VPC egress.
Supported Trigger Types
Trigger
Source
Event or request type
Operational notes
HTTP
Function URL or load balancer
HTTP request
Use Cloud IAM or signed OIDC identity tokens for private endpoints.
Cloud Pub/Sub
Topic
google.cloud.pubsub.topic.v1.messagePublished
At-least-once delivery, retries, and dead-letter topics are supported.
Cloud Storage
Bucket
Object finalized, deleted, archived, or metadata updated
Filter by bucket and object-name pattern.
Firestore
Database
Document created, updated, or deleted
Events have no ordering guarantee.
BigQuery
Table
google.cloud.bigquery.table.v1.rowInserted
Delivered through Eventarc.
Other Eventarc events
Audit Logs, Cloud Run, GKE, direct events, and supported partner sources such as AWS S3 and Azure Blob Storage
Provider-specific CloudEvent types
Not every arbitrary CloudEvent is supported.
Cloud Scheduler and Cloud Tasks are invocation patterns rather than separate native event-trigger types. They call an HTTP function, or Scheduler can publish to Pub/Sub for a Pub/Sub-triggered function. Cloud Tasks requires an authenticated HTTP target and provides controlled, rate-limited dispatch.
Firebase Functions is a separate Google-managed product with provider-specific triggers and the Firebase Functions SDK; those triggers should not be presented as native Cloud Run functions trigger types.
Production Considerations
Delivery: Assume at-least-once delivery. Make handlers idempotent using a business key or source message identifier, configure Pub/Sub dead-letter topics, and alert on dead-letter depth.
Failures: Treat exceptions and unsuccessful responses as retryable. Move long-running or high-memory work to Cloud Run Jobs or Batch, and use Workflows for multi-step orchestration.
Observability: Use structured logs, Error Reporting, and Cloud Monitoring metrics for invocations, failures, and latency. Trace requests across Pub/Sub, Eventarc, and downstream services.
Security: Use a dedicated least-privilege service account, keep secrets in Secret Manager, and restrict invoker permissions. For protected public endpoints, use an external HTTPS Load Balancer with Cloud Armor.
State and cost: Keep durable state in managed databases or storage with appropriate backups. min-instances reduces cold starts but bills continuously; memory allocation also determines allocated CPU, so monitor compute and logging costs.
Choosing the Right Service
Need
GCP
AWS
Azure
Short event-driven function
Cloud Run functions
Lambda
Functions
Custom container or long-lived service
Cloud Run services
ECS/Fargate
Container Apps
Batch or long-running work
Cloud Run Jobs
ECS RunTask or Batch
Container Apps jobs
Messaging and event routing
Pub/Sub and Eventarc
SNS/SQS and EventBridge
Service Bus and Event Grid
Best Practices
Deploy only 2nd-gen functions; Cloud Functions 1st gen is legacy and no longer accepts new deployments.
Keep handlers small and focused, and move sustained workloads to Cloud Run services or jobs.
Manage deployments with Terraform’s google_cloudfunctions2_function resource or Cloud Build and Cloud Deploy, and pin supported runtimes.
Roll out revisions gradually, set max-instances, monitor quotas, and test retry, idempotency, and incident-alert paths before production traffic.
23
Which Google Cloud Storage classes fit different access patterns?
KnownRevise
Google Cloud Storage (GCS) is Google Cloud’s managed object-storage service for unstructured data such as images, videos, backups, logs, and data-lake files. Its storage classes determine storage pricing, retrieval behavior, and minimum storage duration; they do not change the object’s location or GCS’s 11-nines durability target.
Bucket: A globally unique container belonging to a Google Cloud project.
Object: A file and its metadata, accessed through a path such as gs://example-org-data-12345/reports/2025.csv.
Locations: A bucket can use a single region, dual-region, or multi-region location. These affect latency, availability, and cost.
Consistency: GCS provides strong consistency for object and metadata operations, so applications can immediately read newly written data.
Interfaces: Objects are available through gcloud storage, JSON and XML APIs, client libraries, and other compatible command-line tools.
Storage Classes
Class or policy
Typical access pattern
Duration and behavior
Standard
Frequent or unpredictable access
No minimum duration; immediate access for active applications, websites, and analytics inputs
Nearline
Accessed about monthly or less
30-day minimum; typically millisecond retrieval for backups and infrequently accessed reports
Coldline
Accessed quarterly or less
90-day minimum; typically millisecond retrieval for long-lived, rarely accessed data
Archive
Rarely accessed, such as legal, media, or disaster-recovery archives
365-day minimum; data is still available in milliseconds, but retrieval and operation charges are the highest
Autoclass
Unpredictable or changing access patterns where object age is a poor proxy
Automatically transitions objects among Standard, Nearline, Coldline, and (when the terminal class is Archive) Archive according to access; no minimum storage durations, early-deletion or retrieval fees, but a per-object management fee applies
Deleting an object before its minimum duration can incur an early-deletion charge. Standard is the default class, but an ordinary bucket can define another default and uploads can specify a class. An Autoclass-managed bucket controls transitions automatically.
Lifecycle and Cost Behavior
A lifecycle policy can automatically transition an object to a colder class as it ages. A transition changes the object’s storage class but does not move it to another bucket or geographic location.
Storage charges depend on bytes stored, storage class, and location.
Nearline, Coldline, and Archive generally charge less per GB stored than Standard but charge more for data retrieval.
Early deletion, repeated reads, bulk retrieval, operation charges, and network egress can materially change total cost.
Archive offers the lowest storage price, but its retrieval and operation charges make frequent reads expensive.
A lifecycle policy based only on age can be inefficient when different objects in the same bucket have very different access patterns; Autoclass can be more suitable in that case.
GCP and Cloud Comparisons
Google Cloud
Closest AWS equivalent
Closest Azure equivalent
When to choose it
Cloud Storage
Amazon S3
Azure Blob Storage
Unstructured objects, event files, backups, media, and data-lake content
Cloud SQL
Amazon RDS/Aurora
Azure Database for PostgreSQL/MySQL
Managed relational databases and SQL transactions
Firestore
Amazon DynamoDB
Azure Cosmos DB
Application documents and real-time NoSQL workloads
Bigtable
Amazon DynamoDB
Azure Cosmos DB for NoSQL
High-volume, wide-column data with predictable access patterns
BigQuery
Amazon Redshift/Athena
Azure Synapse/Fabric
Analytical SQL over large datasets
These service mappings are conceptual. For storage classes, Amazon S3 Standard-IA and Azure Cool most closely resemble GCS Nearline, while S3 Glacier Deep Archive and Azure Archive most closely resemble GCS Archive, although pricing, minimum durations, and retrieval behavior differ.
Production Practices
Enable Uniform Bucket Level Access, public-access prevention, and least-privilege IAM. Use customer-managed encryption keys when regulatory requirements demand them.
Monitor request errors, latency, storage volume, lifecycle behavior, and costs with Cloud Monitoring, Cloud Logging, and budget alerts.
Use bucket versioning, soft delete, or retention policies according to recovery and compliance requirements; versioning alone is not a complete backup strategy.
Test Archive restoration procedures because retrieval charges and egress volume affect incident-recovery cost and duration.
Reclassify only after measuring access frequency. Incorrect lifecycle assumptions often cost more through retrieval and egress charges than the storage savings.
Audit access through Cloud Audit Logs, protect cross-region copies with appropriate IAM, and rehearse restoration before relying on backups during an incident.
24
How do Cloud Storage object lifecycle rules work?
KnownRevise
Object lifecycle rules in Google Cloud Storage automatically transition eligible objects to a specified storage class, delete eligible object versions, abort incomplete multipart uploads, or apply holds. They reduce manual cleanup and storage costs, but their execution is asynchronous rather than an exact real-time deadline.
Read full answerHide full answer
Applies toGoogle Cloud Storage (current stable) and Google Cloud CLI (current stable)
How Lifecycle Rules Work
Lifecycle rules belong to a bucket and can affect both existing and future objects. Each rule contains a condition and one action:
All conditions within a rule are combined with AND.
Separate rules are evaluated independently.
Cloud Storage evaluates rules asynchronously, so an object may remain eligible after its configured age or date.
Rules operate on object versions when versioning is enabled. age is measured from object creation; daysSinceNoncurrentTime measures time since a version became noncurrent.
Cloud Storage does not provide a separate lifecycle configuration on each object. For per-object timing, set the object’s customTime and use a bucket rule containing customTime or daysSinceCustomTime. Per-object temporaryHold and eventBasedHold fields provide separate hold controls.
Conditions and Actions
Common conditions include:
Category
Conditions
Creation and age
age, createdBefore
Custom time
customTime, daysSinceCustomTime
Version status
isLive, numNewerVersions
Noncurrent age
noncurrentTimeBefore, daysSinceNoncurrentTime
Selection
matchesPrefix, matchesSuffix, matchesStorageClass
Common actions are:
Delete: Deletes matching live or noncurrent object versions, subject to versioning, soft delete, retention, and holds.
SetStorageClass: Sets the specified target storage class. It is normally used to move objects into colder tiers such as NEARLINE, COLDLINE, or ARCHIVE.
AbortIncompleteMultipartUpload: Removes multipart uploads that were initiated but never completed.
SetTemporaryHold: Places a temporary hold that must be explicitly removed.
SetEventBasedHold: Places an event-based hold that is removed when the object is replaced.
Production Behavior and Trade-offs
Versioning: Deleting a live object in a versioned bucket normally makes it noncurrent instead of permanently removing it. Use isLive: false when targeting noncurrent versions.
Soft delete: With soft delete enabled, lifecycle deletion produces a recoverable soft-deleted object for the configured period.
Holds and retention: Existing holds can block lifecycle deletion. Temporary holds block deletion and replacement until removed; event-based holds are cleared by replacement. Retention policies provide stronger compliance protection.
Storage economics: Nearline, Coldline, and Archive transitions can incur minimum-duration charges and retrieval fees. Archive retrieval can also have substantial latency.
Recovery and monitoring: Lifecycle rules are not backups. Use versioning, replication, or a dedicated backup strategy, and monitor object counts, bytes by storage class, lifecycle-related activity, and bucket configuration changes in Audit Logs.
AWS and Azure Equivalents
Capability
Google Cloud Storage
Amazon S3
Azure Blob Storage
Age and date expiration
Lifecycle age and date conditions
Lifecycle expiration
Lifecycle management policies
Prefix and tag filters
Prefix and suffix filters
Prefix and tag filters
Prefix and blob index tag filters
Storage transitions
SetStorageClass
Storage-class transitions
Access-tier and Blob-tier transitions
Per-object timing
customTime
No direct custom-timestamp equivalent
No direct custom-timestamp equivalent
Code Example
This configuration transitions objects under logs/archive/ to Nearline after 30 days, deletes objects under logs/temp/ after 30 days, and aborts incomplete multipart uploads after one day.
Apply and inspect the configuration with the gcloud CLI:
# Replace the lifecycle configuration on an existing bucket.
gcloud storage buckets update gs://example-bucket \
--project=example-project \
--lifecycle-file=./lifecycle.json
# Inspect the effective bucket configuration.
gcloud storage buckets get gs://example-bucket \
--project=example-project \
--format=json
Best Practices and Common Pitfalls
Test rules on a nonproduction bucket and use narrow prefixes or suffixes.
Treat the lifecycle file as the desired complete configuration because updating the bucket’s lifecycle configuration can replace existing rules.
Use versioning, soft delete, retention, holds, and replication as complementary controls rather than assuming lifecycle deletion guarantees immediate removal.
Grant only the required IAM permissions, including storage.buckets.update for lifecycle changes and storage.objects.update for per-object metadata or hold changes.
A signed URL in Google Cloud Storage is a time-limited HTTPS URL that grants temporary access to a specific object without exposing the service account private key or requiring an OAuth bearer token. The complete URL is itself a bearer credential; V4 is recommended, while V2 remains supported for compatibility.
Read full answerHide full answer
Applies toGoogle Cloud Storage signed URLs (V4 recommended; V2 supported for compatibility), current Google Cloud; gcloud CLI (current stable)
How It Works
For V4 signing, a service account signs a canonical HTTP request with its signing key.
The URL contains X-Goog-Algorithm, X-Goog-Credential, X-Goog-Date, X-Goog-Expires, X-Goog-SignedHeaders, and X-Goog-Signature.
The signature covers the HTTP method, canonical object path, relevant query parameters, and designated headers.
Cloud Storage validates the signature and expiration and authorizes access according to the signing service account’s applicable IAM or ACL permissions.
V4 signed URLs can remain valid for up to seven days. Short expirations substantially reduce the impact of disclosure.
A signed URL does not make the bucket or object public and does not override applicable access controls, retention policies, legal holds, lifecycle behavior, or CMEK checks.
For example, a signed GET URL permits a browser to download reports/example.pdf. A signed PUT URL can authorize a direct upload when the signing service account has storage.objects.create, such as through roles/storage.objectCreator.
Operational Use
The current gcloud storage buckets sign-url command supports IAM-based V4 signing through --service-account; for example: gcloud storage buckets sign-url gs://example-bucket/reports/example.pdf --duration=15m --service-account=signer@example-project.iam.gserviceaccount.com --project=example-project --version=V4.
The principal running gcloud needs iam.serviceAccounts.signBlob on the target service account, commonly granted through roles/iam.serviceAccountTokenCreator on that specific service account. The signing identity also needs object access: scope storage.objects.get to the object for downloads, or grant storage.objects.create for uploads.
Signed URLs do not enable cross-origin browser access by themselves. Configure the bucket’s CORS policy separately when a browser uploads directly to Cloud Storage.
Cloud Comparisons
Cloud
Equivalent
Main difference
Google Cloud Storage
Signed URL
Query-string signature based on a Google service account
Amazon S3
Presigned URL
Similar temporary capability generated through AWS credentials or SDKs
Azure Blob Storage
Shared Access Signature
SAS tokens can contain broader permissions and service policies
All three provide temporary delegated access. The principal operational concern is the same: minimize the lifetime, protect the complete URL, and monitor its use.
Security and Failure Modes
Symptom
Likely cause
Corrective action
403 Permission denied
The signing identity lacks object access, or its applicable access has been revoked
Grant only the required object permission and verify current access controls
400 Invalid signature
The URL, HTTP method, path, query, or signed header was changed
Send the URL exactly as generated and use the intended method
400 Request has expired
The URL expired or system clocks are skewed
Use a short lifetime and maintain accurate NTP synchronization
404 Not Found
The object does not exist at the signed path
Verify the bucket, object name, and any generation requirement
403 involving CMEK
The signing identity lacks the required Cloud KMS permission on the key version
Grant the appropriate KMS encrypt/decrypt role and verify key access
Treat a signed URL as a bearer credential. Anyone who obtains it can replay it until it expires, subject to the encoded method and access controls.
Production Practices
Prefer short expirations, commonly 5–15 minutes for downloads, rather than seven-day URLs by default.
Use IAM-based signBlob signing instead of storing service account private keys in CI systems, containers, or secret files.
Grant roles/iam.serviceAccountTokenCreator only on the service account being used for signing.
Never log complete signed URLs in application logs, traces, analytics payloads, support tickets, or browser telemetry.
Enable appropriate Cloud Audit Logs and monitor elevated 400 and 403 rates, unusual download volume, and repeated replay from unexpected networks.
Use narrowly scoped permissions such as storage.objects.get for downloads and storage.objects.create for uploads; avoid bucket-wide viewer grants unless listing is also required.
Prefer uniform bucket-level access and IAM, while accounting for legacy buckets that still use object ACLs.
After revoking access or handling an incident, verify that old URLs are rejected before issuing replacements.
Configure cache and object-generation behavior carefully because URL expiration and object replacement are separate concerns.
26
How do you secure a Google Cloud Storage bucket?
KnownRevise
A secure Google Cloud Storage bucket combines uniform bucket-level access, public access prevention, least-privilege IAM, encryption, retention, and audit monitoring. Treat the bucket as a production data boundary rather than relying on individual object permissions or obscurity.
Read full answerHide full answer
Applies toGoogle Cloud Storage (current stable) and Google Cloud CLI (current stable)
Core Security Controls
Uniform bucket-level access: Use bucket IAM for all access and disable per-object ACLs. This simplifies auditing and prevents inconsistent permissions.
Public access prevention: Block allUsers and allAuthenticatedUsers, even if someone adds a public IAM binding.
Organization policy: Enforce constraints/storage.publicAccessPrevention across the organization so project owners cannot weaken it.
TLS in transit: GCS uses TLS for HTTPS requests; do not permit unencrypted traffic to the storage.googleapis.com API.
Encryption at rest: Google-managed encryption is enabled by default. Use customer-managed encryption keys (CMEK) when key rotation, separation of duties, or key revocation must be centrally controlled.
Private sharing: Use short-lived signed URLs instead of making objects public.
Control
Google Cloud Storage
AWS S3
Azure Blob Storage
Authorization
Bucket/object IAM
Bucket policy and IAM
Azure RBAC
Block public access
Public Access Prevention and organization policy
Block Public Access
Disable Allow blob public access; keep container access private
Per-object ACLs
Disabled with uniform bucket-level access
Generally discouraged
Available but generally discouraged
Customer-managed keys
CMEK
SSE-KMS
Customer-managed keys
Least-Privilege IAM
Grant only the required predefined role, such as roles/storage.objectCreator or roles/storage.objectViewer.
Use a custom role when the separation between read, upload, overwrite, and delete permissions matters.
Prefer workload identities for GKE, Cloud Run, and other Google Cloud workloads. Avoid downloadable service-account keys.
Do not grant roles/storage.admin or roles/owner merely to let an application upload files.
Remember that bucket-level object roles can grant access to every object in the bucket. Use separate buckets for different trust boundaries when isolation is required.
Data Protection and Recovery
Enable object versioning to recover from accidental overwrite or deletion.
Configure lifecycle rules to delete obsolete noncurrent versions before they create unexpected storage costs.
Use retention policies or holds for regulated data. A bucket-level retention lock is irreversible and prevents shortening or removing the retention period.
Verify the bucket’s soft-delete configuration; soft delete and versioning serve different recovery purposes.
CMEK does not replace backups. For critical data, replicate to a separately administered bucket or region, protect the destination independently, and test restoration procedures.
Keep sensitive secrets in Secret Manager rather than storing them as ordinary bucket objects.
Monitoring and Incident Response
Enable Cloud Audit Logs, especially Data Access logs for Cloud Storage, and retain them in a protected log bucket.
Alert on IAM and bucket-policy changes, public-access attempts, unusual download volume, lifecycle changes, and retention-policy modifications.
Review denied requests and unusual service-account activity with Security Command Center or Cloud Logging.
Include bucket policies, IAM bindings, CMEK configuration, lifecycle rules, and retention settings in configuration-as-code and change reviews.
Back up bucket configuration separately; recreating a bucket does not necessarily recreate its IAM policies or retention controls.
Code Example
Create a private, versioned bucket and grant a workload service account narrowly scoped access:
roles/storage.objectCreator permits object creation but does not grant object-read or explicit object-delete permission. It is not a write-once guarantee: a caller that can replace an existing object can change its contents. Use ifGenerationMatch=0 preconditions, retention or holds, or a separate bucket where existing content must not be replaced.
Production Checklist
Enforce public-access prevention at both bucket and organization levels.
Keep uniform bucket-level access enabled.
Grant access to identities, not broad groups, and review bindings regularly.
Use signed URLs with short expirations for controlled downloads.
Enable versioning, lifecycle management, CMEK where required, and tested recovery procedures.
Monitor audit logs and alert on configuration or access-pattern changes.
Never place real credentials in bucket names, policies, examples, or application configuration.
27
What is Google Cloud SQL managed relational database?
KnownRevise
Cloud SQL is Google Cloud’s managed service for relational databases, supporting PostgreSQL, MySQL, and SQL Server. Google manages provisioning, patching, backups, monitoring integration, and high availability, while applications connect using standard database protocols.
Managed database lifecycle: Google operates the operating system, database binaries, storage, and routine maintenance instead of requiring you to administer database hosts.
Supported engines: PostgreSQL, MySQL, and Microsoft SQL Server provide familiar SQL, transaction, indexing, and ecosystem support.
High availability: A regional instance maintains a synchronous standby in another zone and can fail over automatically within that region.
Read replicas: Additional instances can serve read-heavy workloads or place data closer to users. Cross-region replicas add latency and cost and do not provide a single automatic multi-region failover endpoint.
Backup and recovery: Automated backups support scheduled recovery, while point-in-time recovery uses database transaction logs to restore to a specific time.
Scaling: Instances can change compute tiers and storage capacity. storage-auto-increase lets disk grow without overwriting the existing disk size, within service limits.
Cloud SQL is primarily a transactional database. It is not a globally distributed database like Spanner, a document database like Firestore, or an analytics warehouse like BigQuery.
Production Architecture and Operations
A typical production deployment uses:
A regional high-availability instance for the primary database.
Private IP connectivity through VPC peering, Private Service Connect, or the legacy private-services-access model.
A connection pooler such as the Cloud SQL connectors to prevent excessive connections from exhausting the database’s connection limit.
Read replicas for reporting or geographically distributed reads, with applications checking replication lag.
Cloud Monitoring dashboards and alerts for CPU, memory, disk utilization, storage growth, replication lag, connection counts, and instance availability.
Google performs routine minor-version maintenance during a configurable maintenance window. Major engine-version upgrades generally require you to initiate and validate the upgrade. Before any major upgrade, test compatibility, take a backup, and rehearse the process in a nonproduction environment.
Security and Failure Modes
IAM controls who can administer instances through the Cloud SQL Admin API. Grant narrowly scoped roles such as roles/cloudsql.admin or roles/cloudsql.client rather than broad project roles.
Encryption at rest is provided by default, and customer-managed encryption keys are available for applicable configurations.
TLS protects connections when the client enforces certificate validation.
Private IP avoids exposing the database directly to the internet. Use it whenever the application and database share connected network resources.
Database authentication is separate from Google Cloud IAM; engine-specific IAM authentication should be enabled where supported.
Important failure modes include reaching a connection limit, running out of storage, excessive replication lag, backup failures, and application deployments that are incompatible with a schema change. Capacity alerts, tested restores, and connection pooling help prevent these issues from becoming incidents.
Service Selection
Requirement
Recommended GCP service
Managed relational database with PostgreSQL, MySQL, or SQL Server
Cloud SQL
Globally distributed, strongly consistent SQL with horizontal scaling
Spanner
NoSQL document workloads
Firestore
Wide-column NoSQL workloads
Bigtable
Large-scale analytics and data warehousing
BigQuery
The closest AWS equivalents are Amazon RDS and Aurora; Azure equivalents include Azure Database for PostgreSQL, Azure Database for MySQL, and Azure Database for SQL Server. Choose Cloud SQL when you want a conventional relational engine without server administration. Choose Spanner when global distribution and horizontal scale are primary requirements.
Best Practices and Common Pitfalls
Enable high availability for production workloads that cannot tolerate a zone failure.
Use private networking, TLS, and least-privilege IAM.
Keep application and database capacity aligned; monitor disk headroom and connection saturation.
Test point-in-time restoration regularly rather than assuming backups are usable.
Use read replicas intentionally, and ensure applications tolerate replication lag.
Include Cloud SQL charges for compute, storage, backups, replicas, and network egress in cost forecasts.
Avoid presenting a cross-region read replica as a transparent multi-region failover solution.
28
How do Cloud SQL high availability and read replicas work?
KnownRevise
High availability in Google Cloud SQL uses a synchronous primary–standby pair within one region, while ordinary read replicas provide asynchronously replicated copies for read scaling or manual cross-region disaster recovery. Enterprise Plus cross-region replicas are a separate standby-style disaster-recovery feature that can support automated regional failover; these mechanisms solve different failure modes and do not replace backups.
Read full answerHide full answer
Applies toGoogle Cloud SQL current stable (gcloud CLI current stable; PostgreSQL 16 examples)
Architecture Choices
Capability
Regional high availability
Ordinary read replica
Backup
Replication
Synchronous within a region
Asynchronous; same or different region
Scheduled backups plus transaction logs for PITR
Main purpose
Survive an instance or zonal failure
Scale reads or support manual regional DR
Recover from deletion, corruption, or logical errors
Writes
Primary is writable
Read-only until promoted
Restored copy is writable
Failover
Automatic to the regional standby
Manual promotion
Manual restore
Recovery point
Near-zero under normal HA operation
Depends on replication lag
Depends on backup retention and PITR window
Typical use
Production transactional database
Reporting, analytics, geographic DR
Recovery and compliance
A Cloud SQL primary can have multiple ordinary read replicas. A same-region read replica remains asynchronous: it does not become an automatic failover pair with its source.
How It Works
Regional HA creates a primary and standby instance in the same region. The standby receives synchronous transaction-log updates, allowing Cloud SQL to promote it automatically when the primary becomes unavailable. This protects against instance and zonal failures, but not a complete regional outage.
Ordinary read replicas receive updates asynchronously. Applications connect to a replica’s separate endpoint and direct read-only traffic there. Because replication is asynchronous, the replica may lag behind the primary; even a measured lag of zero does not provide transaction-level point-in-time recovery.
For manual regional disaster recovery, create an ordinary cross-region read replica and promote it after a declared outage. Promotion makes the replica an independent writable instance. The original primary must then be fenced or otherwise prevented from accepting conflicting writes.
Enterprise Plus editions also offer cross-region replicas, a standby-style DR feature distinct from ordinary read replicas. They can be configured for automated regional failover. Replication remains asynchronous, so failover can lose transactions that have not yet reached the replica. Use ordinary read replicas when manual promotion and independent read endpoints are desired; use Enterprise Plus cross-region replicas when automatic DR failover is required.
Code Example
The following commands create a regional HA primary and an ordinary asynchronous cross-region read replica:
# Create a regional HA primary with scheduled backups.
gcloud sql instances create orders-primary \
--project=example-project \
--region=us-central1 \
--database-version=POSTGRES_16 \
--tier=db-custom-4-8192 \
--availability-type=regional \
--backup-start-time=03:00
# Create an ordinary asynchronous cross-region read replica.
gcloud sql instances create orders-read-replica \
--project=example-project \
--region=us-east1 \
--database-version=POSTGRES_16 \
--tier=db-custom-2-4096 \
--master-instance-name=orders-primary
# Promote the ordinary read replica during a regional incident.
gcloud sql instances promote-replica orders-read-replica \
--project=example-project
Promotion is a significant operational action. It is not the automated failover mechanism used by regional HA or configured Enterprise Plus cross-region replicas.
Production Operations
Monitor replication lag: Alert on the Cloud SQL replication-lag metric, CPU, memory, disk utilization, connection saturation, and instance or replica state.
Track operations: Use gcloud sql operations list --instance=orders-primary --project=example-project to follow failovers, promotions, backups, and configuration changes.
Plan failover behavior: Regional HA and automated cross-region failover interrupt existing sessions. Applications need connection retries with exponential backoff and jitter and must replace stale pooled connections.
Use replicas deliberately: Route repeatable, potentially expensive queries to ordinary read replicas, but ensure they tolerate slightly stale results.
Maintain backups: Read replicas do not protect against accidental deletion, corruption, or a harmful schema change. Use scheduled backups and, where supported, point-in-time recovery.
Test recovery: Regularly rehearse replica promotion and backup restoration, including application compatibility and DNS or connection-routing changes.
Security and Cost
Prefer private IP connectivity, restrict authorized networks, and avoid exposing database ports publicly.
Require encrypted database connections using the appropriate SSL/TLS settings for the selected engine.
Use least-privilege IAM identities for operators and automation. Grant the instance service account only permissions required by enabled integrations, such as customer-managed encryption keys.
Data is encrypted at rest by default; Cloud SQL also provides customer-managed key options where required.
Regional HA adds standby capacity, while read replicas add storage and compute resources. Monitor both budget and provider pricing before adding replicas.
Cross-region replicas improve disaster recovery, but they do not create a synchronous multi-region database and therefore do not provide zero-loss regional failover.
Failure Modes and Best Practices
Replica lag: Large writes, indexing, maintenance, or network problems can increase lag. Avoid placing latency-sensitive workflows behind replicas unless stale reads are acceptable.
Promotion conflict: After promoting an ordinary cross-region read replica, ensure the old primary cannot continue serving writes.
Connection routing: An ordinary read replica has its own endpoint. Use explicit routing rather than assuming replicas share the primary’s address.
Resource contention: Cap replicas appropriately and separate reporting workloads from OLTP workloads to avoid degrading the primary.
Schema changes: Apply changes through a controlled migration process and verify compatibility with every replica before completion.
Recovery ownership: Document who declares regional failure, promotes an ordinary replica or enables automated failover, updates routing, and reconciles the former primary.
29
How does Cloud Spanner differ from Cloud SQL?
KnownRevise
Cloud Spanner is Google Cloud’s globally distributed relational database, designed for horizontal scaling and strong consistency across regions. Cloud SQL is a managed relational database service for MySQL, PostgreSQL, and SQL Server, offering familiar database engines with primarily vertical scaling and replica-based high availability.
Read full answerHide full answer
Applies toGoogle Cloud Spanner and Google Cloud SQL (current versions)
How Cloud Spanner Works
Distributed storage: Data is automatically partitioned into tablets and distributed across machines. Applications address one logical database rather than manually managing shards.
Replication: Replicas provide durability and availability. Regional configurations span zones; multi-region or dual-region configurations spread replicas across locations.
Consistency: Spanner provides strongly consistent reads and serializable read-write transactions by default, including in multi-region deployments.
Transactions: Distributed transactions can encounter contention and may need application-level retries, especially when many transactions modify related rows.
Schemas and SQL: It supports relational schemas, indexes, joins, and transactions through the GoogleSQL and PostgreSQL dialects. It is not wire-compatible with MySQL or PostgreSQL servers.
Time and coordination: Internally, Spanner uses consensus protocols and TrueTime to coordinate replicated state and provide external consistency.
How Cloud SQL Works
Cloud SQL runs a managed MySQL, PostgreSQL, or SQL Server instance on Google-managed infrastructure.
Scaling: CPU and memory are increased by moving to larger machine types. Additional read replicas support read-heavy workloads, but a primary instance remains the write bottleneck.
High availability: Regional high-availability configurations maintain a standby in another zone within the same region. Cross-region replicas provide disaster recovery but normally replicate asynchronously.
Compatibility: Existing applications can often migrate with minimal SQL changes, including dialect-specific extensions and operational tools.
Management: Google handles infrastructure, patching, backups, monitoring, and routine maintenance while administrators retain database configuration and engine-level control.
Side-by-Side Comparison
Area
Cloud Spanner
Cloud SQL
Database model
Globally distributed relational database
Managed MySQL, PostgreSQL, or SQL Server
Main scale direction
Horizontal, automatic table splitting
Primarily vertical, with read replicas
Availability model
Regional, multi-region, or dual-region
Regional HA plus optional cross-region replicas
Consistency
Strong consistency and serializable transactions by default
Engine-specific transaction and replication behavior
SQL compatibility
GoogleSQL or PostgreSQL dialect
Native MySQL, PostgreSQL, or SQL Server
Global write latency
Higher in multi-region configurations, but globally available
Cross-region writes depend on asynchronous replicas
Typical workloads
Global transactions, high-volume OLTP, low-latency services
Existing enterprise applications, general OLTP, smaller or regional systems
Closest AWS equivalent
Aurora is relational but not fully equivalent
Amazon RDS or Amazon Aurora
Closest Azure equivalent
Azure SQL Hyperscale is conceptually close
Azure Database for PostgreSQL, Azure Database for MySQL, or Azure SQL Managed Instance
Production Operations
Monitoring: Monitor Spanner transaction latency, contention, CPU, storage, and regional performance. For Cloud SQL, track CPU, disk utilization and I/O, connections, replication lag, and backup status. Use Cloud Monitoring dashboards and alerts in both cases.
Backups: Both support backups and point-in-time recovery. Test restoration regularly; a successful backup job does not prove that the restore procedure meets its recovery-time objective.
Upgrades: Cloud SQL engine upgrades and maintenance windows require application testing, especially for major PostgreSQL, MySQL, or SQL Server versions. Spanner schema and query changes should also be tested for compatibility and performance.
Incidents: Spanner multi-region deployments improve regional resilience, but contention or hot partitions can still affect latency. Cloud SQL failovers can interrupt active connections, so applications need reconnect and retry logic.
Security: Use Cloud IAM with least-privilege roles, avoid static credentials, and use private connectivity where required. Cloud SQL IAM database authentication is available for PostgreSQL and MySQL, but not SQL Server.
Cost: Spanner costs depend on the instance configuration, processing capacity, storage, and network traffic. Cloud SQL costs include instance resources, storage, backups, network usage, and potentially SQL Server licensing.
Useful read-only inventory commands are gcloud spanner instances list --project=example-project and gcloud sql instances list --project=example-project. Backup state can be inspected with gcloud spanner backups list --instance=example-instance --project=example-project and gcloud sql backups list --instance=example-db --project=example-project.
When to Choose Each
Choose Cloud Spanner when global availability, elastic horizontal growth, strong consistency, and high write throughput are primary requirements. Choose Cloud SQL when compatibility with an existing database engine, predictable infrastructure-based sizing, or lower migration effort matters more.
Validate Spanner against the target workload using representative data volume, transaction contention, and multi-region latency. Do not select it solely from benchmark results: global writes, hot partitions, and retry behavior can materially change its operational profile.
30
When is Cloud Firestore the right database choice?
KnownRevise
Google Cloud Firestore is a fully managed, NoSQL document database for storing and synchronizing application data at scale. It is a strong choice when applications need flexible document models, low-latency reads and writes, real-time updates, or offline-first mobile and web experiences.
Documents: Firestore stores records as JSON-like documents containing fields, nested maps, lists, and timestamps.
Collections and subcollections: Documents live in top-level collections and can contain subcollections. Subcollection IDs are unique only within their parent document.
Scalability: The service automatically distributes data and scales capacity, so teams do not manage servers, replicas, or database patching.
Client access:
Server applications typically access Firestore through client libraries.
Firebase client SDKs can connect directly to Firestore, providing real-time listeners and offline synchronization.
Modes:
Native mode is the standard choice for new document-oriented applications.
Datastore mode supports legacy Datastore applications and migration scenarios.
Firestore does not enforce joins, foreign keys, or arbitrary relational constraints. A document should generally represent a cohesive aggregate rather than a small portion of a heavily normalized schema.
Consistency and Querying
Reads of individual documents are strongly consistent.
Transactions provide serializable, strongly consistent access to multiple documents.
A regular multi-document query is not a database-wide snapshot, so applications must tolerate concurrent changes.
Firestore automatically indexes single fields, but many multi-field and collection-group queries require composite indexes.
Security rules are evaluated per document. A query must return only documents the caller is allowed to read; rules do not silently filter unauthorized documents from an otherwise authorized query.
Choosing the Right GCP Service
GCP service
Best fit
AWS/Azure approximation
Firestore
Document data, mobile synchronization, real-time application state, serverless APIs
Amazon DynamoDB or Azure Cosmos DB, though their data models differ
Amazon RDS/Aurora or Azure Database for PostgreSQL/MySQL
Spanner
Large-scale relational data, global distribution, strong transactional guarantees
No exact equivalent; DynamoDB has a different data model
Bigtable
Massive key-value workloads and high-throughput ingestion
Amazon DynamoDB or Azure Table Storage
BigQuery
Analytics, large scans, historical reporting
Amazon Redshift/Athena or Azure Synapse/Fabric
Choose Firestore for flexible application documents and user-facing synchronization. Choose Cloud SQL when relationships and SQL reporting dominate. Choose Spanner for globally distributed, strongly transactional relational workloads. Choose Bigtable for extreme key-value scale, and BigQuery for analytical processing.
Production Behavior
Locations: Firestore offers regional and multi-region locations. Select one deliberately because changing the location of an existing database is generally not supported. Multi-region provides wider geographic availability; regional locations can reduce latency and meet residency requirements.
Reliability: Firestore manages replication, failover, hardware maintenance, and service upgrades. Applications still need retry handling with exponential backoff for transient errors.
Backups: Configure and test the backup and point-in-time recovery features appropriate to the recovery objective. A backup is not useful until a restore has been exercised.
Monitoring: Track client latency, request errors, denied operations, quota consumption, database usage, and billing through Cloud Monitoring and Cloud Audit Logs.
Scaling limits: A single frequently updated document can become a contention point. Partition hot data across multiple documents or move the workload to Bigtable when its scale characteristics differ.
Diagnostics: Inspect the active database with gcloud firestore databases describe --database="(default)" --project="example-project".
Security, Reliability, and Cost Best Practices
Use IAM for service-to-service access and Firestore Security Rules for direct client access. Privileged server SDKs generally use IAM and bypass end-user rules.
Deny access by default, test rules against representative requests, and never store credentials or sensitive secrets inside documents.
Place sensitive fields behind server-side APIs when direct client access is unnecessary.
Keep documents reasonably sized and avoid unbounded lists. Firestore charges for reads, writes, storage, and network transfer, so document fan-out and chatty queries can increase cost sharply.
Use transactions for related invariants, but do not place large workloads inside long-running transactions.
Set alerts for elevated errors, latency, quota pressure, and unusual read or write volume. Review quota limits before traffic events or major launches.
31
What is Cloud Bigtable and which workloads suit it?
KnownRevise
Cloud Bigtable is Google Cloud’s managed, NoSQL wide-column database for very large tables that require low-latency reads and high-throughput writes. It implements the original Google Bigtable data model and is suited to applications with a well-understood access pattern rather than ad hoc relational or document queries.
Read full answerHide full answer
Applies toGoogle Cloud Bigtable current GA (Data API, HBase API, and emulator)
Architecture and Capabilities
Data model: A table contains rows identified by row keys, column families, column qualifiers, and timestamped cells. Cell versions remain until a garbage-collection policy removes them; scans expose row-key order, not insertion order.
Transactions: Bigtable provides strongly consistent reads and writes, single-row atomicity, and conditional mutations. It does not provide multi-row or cross-table transactions.
Scale and availability: Tables are split into tablets and automatically rebalanced. Three-way replication provides node-level availability, while multi-region instances can place replica clusters in different regions.
Interfaces: The primary Data API supports gRPC and REST. Bigtable also offers the HBase API, language-specific client libraries, Change Streams, authorized views, and a local emulator for development and tests.
Query limitations: There is no general-purpose SQL layer or arbitrary native secondary-index mechanism. Supported index-shaped access patterns can use materialized views; other cases require a deliberately designed reverse table.
Suitable and Unsuitable Workloads
Telemetry and time series: Device or monitoring data can be partitioned by entity and time, with wide rows and garbage-collection policies controlling retention.
Event pipelines: Pub/Sub can deliver events into Bigtable for durable serving or downstream processing. Ordering must be handled explicitly because Bigtable does not preserve Pub/Sub ordering by itself.
Feature serving: Recommendation, personalization, and ad-tech systems benefit from predictable key lookups and horizontal scale.
User activity and operational logs: High-cardinality activity data fits the wide-column model when retention and query patterns are planned.
Poor fit: Use Cloud SQL for relational OLTP, Spanner for globally distributed transactional workloads, BigQuery for analytical scans, and Firestore for flexible documents and synchronization. Use Cloud Storage lifecycle tiers for inexpensive long-term archival data.
Comparison With Other Cloud Data Stores
Primary need
Google Cloud
AWS
Azure
Very large keyed tables
Bigtable
DynamoDB
Cosmos DB Table API
Document store and mobile synchronization
Firestore
DynamoDB
Cosmos DB
Relational OLTP
Cloud SQL
RDS or Aurora
Azure Database for PostgreSQL
Serverless analytical scans
BigQuery
Redshift Serverless or Athena
Fabric or Synapse serverless
Production Operations
Capacity: Use provisioned nodes for predictable load and autoscaling for bursty demand. Monitor sustained utilization rather than reacting to isolated spikes.
Key distribution: Avoid hot tablets caused by sequential or skewed keys. Prefixing or salting can spread writes, although it may sacrifice convenient range ordering and must be designed consistently.
Backups: Scheduled backups create point-in-time copies that can be restored to a new instance. Define the recovery point objective from the schedule, retain backups for the required period, and rehearse restores.
Monitoring: Track request latency, error rate, node utilization and count, backup status, and replication health. Alert on symptoms that affect users rather than relying only on infrastructure metrics.
Upgrades and incidents: Google manages the service’s infrastructure and rolling upgrades, but clients should still use bounded exponential backoff for transient UNAVAILABLE and RESOURCE_EXHAUSTED responses. A regional instance is vulnerable to a regional outage; use a multi-region instance with replica clusters and an appropriate routing policy, such as ANY_CLUSTER, when regional fault tolerance is required.
Security, Cost, and Pitfalls
IAM: Use the narrowest supported scope with predefined roles such as roles/bigtable.reader, roles/bigtable.user, and roles/bigtable.admin. Authorized views can expose only selected rows and columns.
Encryption and controls: Encryption at rest is enabled by default, with Cloud KMS customer-managed encryption keys available for applicable requirements. VPC Service Controls adds exfiltration protection; residency must still be enforced through instance, replica, and backup-location policies.
Cost: Provisioned node capacity and stored data are the principal cost drivers. Compare SSD and HDD capacity for the workload, use autoscaling carefully, and review idle tables, backups, and unused replicas.
Quotas and failure handling: Excessive load can produce throttling or increased latency rather than silent data loss. Apply backpressure, cap retries, and avoid full-table scans in latency-sensitive request paths.
32
What is Google Cloud Memorystore?
KnownRevise
Google Cloud Memorystore is a family of managed, in-memory, Redis-compatible data stores for low-latency caching, session storage, leaderboards, and other application state. Google Cloud operates the infrastructure and control plane; applications connect with Redis or Valkey clients, depending on the Memorystore offering.
Read full answerHide full answer
Applies toGoogle Cloud Memorystore current offerings: Memorystore for Redis and Redis Cluster (Redis 7.x), and Memorystore for Valkey (Valkey 8.x)
Core Capabilities
Compatibility: Supports standard Redis clients, common commands, and data structures such as strings, hashes, lists, sets, and sorted sets.
Separate offerings: Memorystore for Redis is a regional, single-primary service with an optional standby replica. Memorystore for Redis Cluster is a sharded, multi-primary service with replicas and automatic failover. Memorystore for Valkey is a separate Valkey-based offering with its own tier, capacity, and regional feature matrix. One instance cannot switch between Redis and Valkey.
High availability: Replicas can be placed across zones within a region, with managed failover when supported by the selected offering.
Scaling: Cluster-based offerings support scaling capacity and replicas, subject to quotas and regional availability.
Managed operations: Google provisions and monitors the service, performs infrastructure maintenance, and integrates with Cloud Monitoring.
Private networking: Applications connect through supported private VPC or Private Service Connect paths rather than a public internet endpoint.
Memorystore is usually a cache or ephemeral application tier. It should not automatically be treated as the authoritative system of record.
How It Works in Production
A common cache-aside request path is:
The application reads from Memorystore.
A cache hit returns the stored value.
On a miss, the application queries Cloud SQL, Cloud Spanner, Firestore, or another source of truth.
The application caches the result and returns it to the client.
Replication improves availability, but it is not equivalent to a backup. Some offerings provide snapshots or backup and restore, but persistence, retention, and recovery behavior differ by product and tier. Teams should verify recovery-point and recovery-time objectives and test restores rather than assuming zero data loss.
GCP and Cloud Comparisons
Requirement
Google Cloud
AWS equivalent
Azure equivalent
Managed in-memory key-value store
Memorystore
ElastiCache
Azure Managed Redis
Durable relational database
Cloud SQL
RDS or Aurora
Azure Database for PostgreSQL or MySQL
Globally distributed transactional database
Cloud Spanner
DynamoDB or Global Aurora
Cosmos DB
Document data model
Firestore
DynamoDB
Cosmos DB
Wide-column NoSQL data
Bigtable
DynamoDB
Cosmos DB for NoSQL
Choose Memorystore when predictable low-latency access is the priority. Choose Cloud SQL or Spanner for transactional durability, and Firestore or Bigtable when their document or wide-column data models fit the workload.
Failure Modes and Security
Memory exhaustion: Depending on the configured memory and eviction behavior, writes may be rejected or cached entries may be evicted.
Failover: Replication can be asynchronous, so recent writes may be lost during a failover. Clients also need bounded reconnect and backoff logic.
Regional failure: Replicas within one region do not by themselves provide regional disaster recovery; use backups and a separate recovery design where required.
Connection storms: A recovering service can be overwhelmed by simultaneous reconnects. Use connection pooling, jitter, and exponential backoff.
Administrative commands: Some administrative operations can block or change server behavior. Use only the controls documented for the selected offering; do not assume that upstream Redis ACL features are available.
Network exposure: Keep endpoints on private networking and restrict access to application workloads.
Identity and credentials: Use IAM for GCP resource administration and the offering’s documented authentication mechanism for data access. Keep credentials in Secret Manager rather than source code.
Encryption: Confirm the encryption and customer-managed key options applicable to the selected offering; they are not uniform across Memorystore products.
Best Practices
Use explicit TTLs for cache entries and select an eviction policy only where the offering supports it.
Monitor memory utilization, evictions or rejected writes, CPU, command latency, connections, replica health, and failover events in Cloud Monitoring.
Test failover, scaling, network interruption, restore, and regional recovery through failure drills.
Maintain capacity headroom for failover and traffic spikes.
Review quotas, cross-zone or data-transfer charges, replica costs, and budget alerts before provisioning.
Review managed release notes, maintenance windows, client compatibility, and major-version behavior before production upgrades.
33
How does BigQuery differ from a traditional data warehouse?
KnownRevise
BigQuery is Google Cloud’s fully managed, serverless data warehouse and analytics platform. Unlike a traditional warehouse, it separates storage from compute and automatically allocates query resources, reducing cluster administration, capacity planning effort, and infrastructure maintenance.
Read full answerHide full answer
Applies toGoogle Cloud BigQuery (current service version; Standard, Enterprise, and Enterprise Plus editions)
How BigQuery Works
BigQuery stores data in Google Cloud’s managed infrastructure using a columnar format optimized for analytical queries. It uses distributed query execution to scan large datasets in parallel.
Its main architectural characteristic is separated storage and compute:
Managed storage: Table data is durably stored and scaled by Google Cloud.
Independent compute: Query processing uses temporary compute resources that can scale independently of data volume.
Elastic execution: Large scans can run across many workers without provisioning warehouse nodes.
Slot-based compute: In capacity-based configurations, a slot represents approximately 1 vCPU and 1 GB of memory. Capacity can be purchased on demand or managed through reservations and autoscaling.
Unlike a cluster-based warehouse, a BigQuery job can still consume substantial compute while waiting for resources. Production teams therefore need workload management, query prioritization, and cost monitoring.
Comparison With Other Warehouses
Area
BigQuery
Traditional data warehouse
Amazon Redshift
Azure Synapse Analytics
Infrastructure
Serverless; no cluster administration
Nodes, storage, and software managed by the customer
Provisioned clusters or managed serverless capacity
Dedicated pools or serverless SQL
Storage and compute
Independently managed
Usually closely coupled
Usually cluster-oriented, with serverless options
Dedicated or serverless compute models
Scaling
Automatically scales query execution
Commonly requires adding nodes or reconfiguring a cluster
Requires scaling clusters or serverless capacity
Requires managing pools or serverless capacity
Pricing
Query processing, storage, or reserved capacity
Primarily provisioned infrastructure and storage
Cluster capacity, serverless usage, or reservations
Compute, storage, and licensing by workload
Administration
Little infrastructure administration
Significant operational responsibility
Moderate to low, depending on the model
Moderate to low, depending on the model
Query characteristics
Large scans, SQL analytics, streaming, and data lake workloads
Broad SQL analytics and reporting
SQL analytics and data lake integration
SQL analytics, data lake, and Spark workloads
BigQuery is conceptually closer to a managed cloud warehouse such as Redshift Serverless or the serverless SQL pool in Azure Synapse than to a self-hosted database. Its differentiator is the depth of serverless integration and automatic separation of compute from storage.
Data Ingestion and Ecosystem
BigQuery supports several ingestion patterns:
Batch loads: Data is imported from Google Cloud Storage, other cloud storage, or supported files.
Streaming inserts: New rows can be inserted continuously, commonly through the BigQuery Storage Write API.
Pub/Sub integration: A Pub/Sub subscription can write records into BigQuery through the Storage Write API.
Dataflow pipelines: Dataflow provides managed batch and streaming pipelines for transformations and BigQuery writes.
External and federated queries: BigQuery can query supported data in Cloud Storage and perform cross-cloud or external-data federation with appropriate configuration.
For high-volume ingestion, the Storage Write API is generally more operationally appropriate than issuing individual streaming-insert requests. A common production architecture uses Pub/Sub for event delivery, Dataflow for validation and transformation, and BigQuery for analytical storage.
Production Operations
BigQuery manages infrastructure concerns such as node failures, storage expansion, and software upgrades, but the underlying service can still experience outages and capacity constraints. Application and workload failures also remain possible.
Query failures: Queries can fail because of invalid SQL, resource contention, exceeded limits, quota violations, or upstream data problems.
Long-running queries: Monitor elapsed time, bytes processed, slot usage, and reservation contention.
Observability: Use Cloud Monitoring, Cloud Logging, BigQuery audit logs, INFORMATION_SCHEMA, and job metadata.
Cost control: Partition large tables, use clustering selectively, limit unnecessary columns, and apply table or partition expiration where retention permits.
Recovery: Time travel supports recovery of recently changed or deleted table data. Table snapshots provide explicit point-in-time recovery.
Regional resilience: Choose an appropriate region or multi-region, and design critical analytical workloads around service availability and recovery objectives.
Data lifecycle: Expiration policies reduce cost, but deleting expired data can also remove access needed for incident investigation or compliance.
Storage and compute are separated, but this does not eliminate capacity planning. Heavy concurrency can still require reserved capacity to provide predictable latency and cost.
Security and Governance
BigQuery supports Google Cloud IAM, dataset-level access, and predefined or custom roles. Use the least-privilege role that grants the required table and job permissions; avoid assigning broad project-level owner or editor roles to analytics users.
Key controls include:
IAM authorization: Grant roles/bigquery.dataViewer or more specific permissions only where required.
Column and row protection: Use authorized views, row-level security policies, policy tags, and dynamic data masking where supported.
Encryption: Google Cloud encrypts data at rest and in transit; customer-managed encryption keys can be used where regulatory requirements justify them.
VPC Service Controls: Helps prevent data exfiltration around Google Cloud resources and APIs.
Auditing: BigQuery audit logs support investigations into data access and administrative activity.
Workload identity: Prefer attached service accounts over downloadable service-account keys.
For sensitive datasets, apply classification, retention, deletion, and access-review processes. IAM controls who may query data, but it does not replace data-quality validation or governance of downstream exports.
When to Choose BigQuery
Choose BigQuery when the workload involves large-scale SQL analytics, variable demand, ad hoc exploration, reporting, streaming ingestion, or querying data in Cloud Storage.
Do not treat it as an OLTP database: high-frequency transactional reads and writes usually fit Cloud SQL, AlloyDB, or Spanner better. Choose a traditional or provisioned cloud warehouse when sustained workloads require fixed low-latency compute, highly specialized tuning controls, or contractual capacity predictability. On Google Cloud, BigQuery is usually the strongest default for analytical workloads unless those operational requirements point elsewhere.
34
How does BigQuery pricing work?
KnownRevise
BigQuery pricing combines query-processing charges with storage and optional feature charges. Standard and Enterprise editions primarily charge per TiB scanned, while Capacity editions charge for baseline and autoscaled slot capacity.
Read full answerHide full answer
Applies toGoogle BigQuery Standard, Enterprise, and Capacity editions (current pricing)
Pricing Models
Model
Primary billing meter
Best suited for
Standard on-demand
Bytes processed; at current published rates, the first 1 TiB per month is free per billing account per region, then $6.25 per TiB
Intermittent or unpredictable analytics
Enterprise on-demand
Bytes processed at $10 per TiB, with no monthly free query-processing allowance
Workloads requiring Enterprise features
Capacity
Baseline capacity and autoscaled slots, priced by slot time rather than bytes processed
Stable, sustained, or high-concurrency analytics
On-demand queries are charged for the data examined, not the number of rows returned. A query that returns no rows can still incur charges if it examines billable data. Selecting fewer columns and applying partition filters usually reduces the bytes processed.
Capacity editions provide committed baseline slots and additional autoscaling for temporary demand. They avoid per-TiB on-demand query charges, but baseline overprovisioning wastes money and excessive autoscaling can materially increase the bill.
Other Charge Dimensions
Storage: Billed per GiB-month. Active, Long-term, and Archival logical storage have different rates and minimum-duration implications; lifecycle policies can reduce storage cost.
Streaming ingestion: BigQuery Storage Write API and streaming inserts are charged separately from query processing, based on the applicable row and volume meters.
Data transfer: Transfers outside Google Cloud can incur standard network charges. Separately metered transfer and analytics products may also charge.
Additional features: BigQuery ML operations and BI Engine capacity can incur charges beyond ordinary query processing.
Scheduled and materialized work: Scheduled queries and materialized-view refreshes consume the same underlying query-processing resources as interactive queries. Materialized-view storage may also incur storage charges.
Reducing Query Costs
Filter partitions: Use a valid partition filter so BigQuery can prune irrelevant partitions.
Use clustering selectively: Cluster on frequently filtered or joined columns so BigQuery can skip unnecessary blocks. Clustering complements rather than replaces partition pruning.
Select required columns: Avoid SELECT *; BigQuery’s columnar storage means unreferenced columns generally need not be read.
Reuse results: Materialized views, destination tables, and scheduled aggregates avoid repeatedly computing the same result, although their refreshes and storage are billable.
Use approximate functions: APPROX_COUNT_DISTINCT is substantially cheaper than exact COUNT(DISTINCT ...), but returns an approximate result.
Set a byte ceiling: maximum_bytes_billed prevents an on-demand job from processing beyond a configured limit.
Control wildcard tables: Explicitly constrain wildcard ranges and use dry-run estimates to detect unexpectedly broad scans. A dry run estimates processed bytes but does not display a resolved partition list.
Match capacity to demand: Use on-demand pricing for spiky workloads and Capacity editions for sustained demand. Monitor utilization before adding baseline slots.
AWS and Azure Equivalents
Requirement
Google Cloud
AWS
Azure
Serverless, pay-per-query analytics
BigQuery on-demand
Amazon Athena
Fabric Warehouse or Synapse serverless
Managed reserved analytics compute
BigQuery Capacity
Amazon Redshift
Fabric capacity or Synapse dedicated pools
Choose BigQuery for a fully managed warehouse spanning Google Cloud data, Athena for S3-native ad hoc analysis, Redshift for more provisioned warehouse control, and Fabric or Synapse where Microsoft-oriented tooling and licensing are priorities.
Production Guardrails and Pitfalls
Monitor total_bytes_processed, total_bytes_billed, query duration, and total_slot_ms in INFORMATION_SCHEMA.JOBS.
Export detailed Cloud Billing data to BigQuery and attribute charges using project, service, and label dimensions. Queries against that export can themselves incur BigQuery processing charges.
Configure billing budgets and alerts, but do not treat alerts as automatic spending controls.
Apply least-privilege IAM and separate projects or teams where stronger cost and data boundaries are required.
Do not assume logical views reduce cost because queries against views can still examine the underlying tables.
Use dry runs and EXPLAIN to validate pruning before production rollout, while retaining byte caps because estimates can differ from actual processing.
35
How do partitioned and clustered tables work in BigQuery?
KnownRevise
Partitioned tables divide data into prunable units, while clustered tables organize data by frequently filtered columns. BigQuery supports both independently and can combine them to narrow a scan first by partition and then by data block.
Partitioning splits a logical table into separate partitions. When a query includes a filter on the partition key, BigQuery can use partition pruning to avoid reading irrelevant partitions.
The supported partitioning approaches include:
Date/time column partitioning: Partitions by a DATE, TIMESTAMP, or DATETIME column, or by a supported expression derived from one. The available partition granularity depends on the source type.
Ingestion-time partitioning: Uses the time when data was ingested. Daily ingestion-time tables expose _PARTITIONDATE, while _PARTITIONTIME identifies the exact ingestion-time partition boundary.
Integer-range partitioning: Assigns integer values to user-defined ranges and is useful for numeric bucketing.
Pruning is most effective when a query filters directly on the partitioning column or on a filter that BigQuery can map to the partition boundary. Applying an unsuitable wrapper function can reduce pruning effectiveness. Partition granularity also affects storage: excessive granularity creates many small partitions and metadata overhead.
For eligible time-based partitions, require_partition_filter = TRUE prevents queries that omit a partition filter. It enforces the presence of a filter but does not by itself guarantee that the filter prunes data efficiently. The partition_expiration_days option can delete old partition data automatically, subject to retention and recovery requirements.
Clustering in BigQuery
Clustering organizes table data according to one to four clustering columns. The first column has the highest precedence, followed by the remaining columns in their declared order.
BigQuery stores clustering metadata with its data blocks. When query predicates apply to clustering columns, BigQuery can skip blocks whose stored value ranges cannot satisfy the predicate. Clustering therefore works best for common equality or range filters, such as customer_id, tenant_id, or status.
Clustering has several important limitations:
It is not a traditional B-tree index.
It does not enforce uniqueness.
It does not guarantee a global or query-result row order.
Its effectiveness depends on query patterns, data distribution, and block pruning.
It does not replace partitioning when the primary requirement is time-based filtering, pruning, or retention.
Partitioning Compared With Clustering
Aspect
Partitioning
Clustering
Purpose
Divide a table into independently prunable units
Organize data by frequently filtered columns
Data model
Logical partitions based on time, ingestion time, or integer ranges
Column-oriented sort order represented through data blocks and metadata
Primary benefit
Time-based pruning and lifecycle management
Block pruning within the data selected by other filters
Typical filters
Partition dates, timestamp ranges, or integer-range boundaries
Equality and range predicates on business columns
Retention
Supports partition expiration for eligible time-based partitions
Does not provide an independent retention mechanism
Relationship to other feature
Can be clustered within each partition
Can be combined with partitioning
Main limitation
Poor granularity or filters that do not prune
Unhelpful columns or highly skewed query patterns
When both are used, BigQuery can first exclude whole partitions and then apply clustering metadata to the remaining data. A partition by event date combined with clustering by customer and event type is a common design, although the clustering columns should be validated against actual query statistics.
Production and Operational Considerations
Cost: Query charges depend on bytes processed or capacity consumption, while storage has separate charges. Effective pruning reduces unnecessary scanning, but excessively granular partitions can increase metadata and management overhead.
Ingestion: Clustering can add data-organization work during writes, so representative batch and streaming loads should be benchmarked. Clustering does not impose a separate clustering charge.
Monitoring: Track bytes scanned, slot time, execution latency, ingestion failures, partition counts, and pruning behavior through Cloud Monitoring and BigQuery metadata or system views.
Schema and workload changes: Reassess partition granularity and clustering columns as data volume, retention needs, and common predicates change. Removing clustering columns can make subsequent writes cheaper because the table no longer needs that organization.
Security: Apply IAM controls at the project and dataset levels. Partitioning and clustering are storage-organization features, not access-control boundaries.
Recovery: Partition expiration is not a backup mechanism. Use time travel or table snapshots for point-in-time recovery, copy data to durable storage for longer-term retention, and configure cross-region disaster recovery when regional failover is required.
Best Practices and Common Pitfalls
Choose a partition column that nearly every production query filters on.
Match partition granularity to the workload; avoid creating tiny partitions unless the operational benefit justifies them.
Prefer direct predicates on the partition column instead of filtering through wrapper functions.
Select clustering columns from observed query filters rather than assumptions.
Keep clustering-column data reasonably distributed and avoid columns with severe skew.
Do not rely on clustering for uniqueness or result ordering.
Verify pruning with a dry run (estimated bytes processed) and the job’s execution details; BigQuery has no EXPLAIN statement.
Re-evaluate the design periodically as data volume, retention requirements, and query patterns change.
36
How does BigQuery ML enable machine learning?
KnownRevise
BigQuery ML is Google Cloud’s SQL-native machine-learning service. It trains models from BigQuery data and performs inference through SQL, allowing analytical models to be built without first exporting warehouse data to a separate training platform.
Read full answerHide full answer
Applies toGoogle Cloud BigQuery ML (all versions)
How It Works
Train with SQL: CREATE MODEL selects an algorithm, identifies the label column, and reads training data from a table or query.
Process the data: BigQuery plans and executes the training query using its managed compute infrastructure.
Store the model: The resulting model and its metadata become BigQuery resources in a dataset.
Evaluate and predict: Functions such as ML.EVALUATE, ML.PREDICT, and ML.EXPLAIN_PREDICTIONS expose evaluation and scoring through SQL.
For models trained directly from BigQuery tables, views, joins, or aggregates, data does not need to be exported to another machine-learning platform. This is particularly effective when the source is already structured as large warehouse tables.
Supported Modeling Approaches
BigQuery ML covers common analytical and tabular workloads:
Supervised learning: Linear regression, logistic regression, boosted trees, and deep neural networks.
Unsupervised learning: K-means clustering.
Recommendation systems: Matrix factorization and factorization machines.
Forecasting: Time-series models.
Evaluation: Metrics such as accuracy, precision, recall, F1 score, RMSE, and algorithm-specific measures, depending on the model type.
Typical uses include churn prediction, demand forecasting, fraud scoring, customer segmentation, and operational classification. Vertex AI is generally preferable when a workload requires custom training code, distributed deep learning, a feature store, a model registry, or a managed real-time endpoint.
Production Behavior
Retraining: BigQuery ML does not automatically retrain a model when its input data changes. Schedule controlled retraining with Cloud Scheduler, Workflows, or another orchestration layer.
Evaluation: Evaluate a candidate model on held-out data before promoting it. A newly trained model is not automatically better merely because it is newer.
Monitoring: Use Cloud Monitoring for BigQuery operational metrics and scheduled queries or dashboards for evaluation results, training cost, data freshness, drift, and prediction distributions.
Scoring: Native BigQuery ML predictions execute as part of a query. BigQuery ML itself does not expose a continuously running prediction endpoint.
Real-time serving: Supported BigQuery ML models can be exported to Cloud Storage and deployed through Vertex AI. That creates a separate Vertex AI serving path with its own scaling, latency, monitoring, and cost implications.
Cost control: Monitor bytes scanned, slot usage, training frequency, and inference-query size. A prediction over a large table can be expensive because it may scan substantial data.
Incident diagnosis: Unexpected predictions can result from stale training data, schema or concept drift, data leakage, changed feature transformations, or model degradation—not only from an infrastructure failure.
Service Comparison
Capability
BigQuery ML
Vertex AI
Amazon SageMaker AI
Azure Machine Learning
Primary strength
SQL-native analytical ML
Full ML and MLOps platform
Broad AWS ML platform
Enterprise Azure ML platform
Typical data source
BigQuery tables and queries
BigQuery, local, cloud, or streaming data
S3 and other AWS sources
Data Lake, Blob Storage, and Azure sources
Training interface
SQL
Python, notebooks, APIs, and SDKs
Python, notebooks, and SDKs
Python, notebooks, and SDKs
Real-time serving
Not provided directly by BigQuery ML
Managed endpoints and other serving options
Managed endpoints
Managed endpoints
Best fit
Embedded warehouse predictions
Custom models, deep learning, and MLOps
Existing AWS-centric teams
Existing Microsoft-centric teams
Production Best Practices
Prevent leakage: Exclude post-outcome fields and train only on observations available before the prediction period.
Validate feature types: Keep training and scoring columns compatible; schema changes can cause query failures or materially alter results.
Compare before replacement: Evaluate candidate models against the current model on the same validation population.
Version the pipeline: Store model-creation SQL, feature definitions, evaluation queries, and the training-data cutoff in source control. Use separate model names or reproducible model artifacts for rollback.
Apply least privilege: Grant training and inference identities only the required BigQuery dataset, model, and table permissions.
Automate promotion gates: Schedule retraining with evaluation thresholds, logging, failure alerts, and an explicit promotion or rollback step.
Monitor drift: Track input distributions, label rates, missing values, and prediction distributions even when model execution itself remains healthy.
37
How does Google Cloud Pub/Sub deliver messages?
KnownRevise
Google Cloud Pub/Sub delivers messages asynchronously through topics and subscriptions: publishers send messages to a topic, and each attached subscription receives its own copy for independent pull or push processing. Delivery is generally at least once, so consumers must handle duplicates and acknowledge messages only after successful processing.
Read full answerHide full answer
Applies toGoogle Cloud Pub/Sub (current service)
How It Works
A producer publishes a message to a topic. The publisher client may batch messages before sending them to the Pub/Sub service.
Pub/Sub makes the message available to every subscription attached to that topic. Each subscription tracks its own acknowledgement state, while multiple consumers attached to one subscription share the messages as competing consumers.
A subscriber receives messages through pull by calling Pub/Sub, or through push when Pub/Sub sends HTTPS POST requests to a configured endpoint.
After successful processing, the subscriber acknowledges the message. A pull subscriber does this with an acknowledgement request; for push delivery, Pub/Sub treats HTTP 102 and 2xx responses as success.
If a message is negatively acknowledged or its acknowledgement deadline expires, Pub/Sub redelivers it. Unacknowledged messages are retained for a bounded period, currently up to seven days, rather than indefinitely.
A message can contain payload data, attributes, a server-assigned message ID, a publish timestamp, and an optional ordering key. A subscription can also have a dead-letter topic. After the subscription’s configured maximum number of failed delivery attempts, the message is forwarded to that topic for investigation and controlled replay; without a dead-letter policy, it is dropped.
Delivery Semantics
At-least-once delivery: A message can be delivered more than once if processing succeeds but the acknowledgement is lost, or if the subscriber fails before acknowledging it.
Idempotent processing: Consumers should make repeated handling safe, commonly by enforcing a unique constraint on the subscription ID plus message ID or an application event ID.
Exactly-once delivery: Pub/Sub supports this only for pull subscriptions with the required publisher and subscription opt-in. It prevents Pub/Sub from redelivering a successfully acknowledged message, but it does not make external side effects atomic with the acknowledgement, so idempotency remains necessary.
Ordered delivery: When the publisher client enables message ordering, messages with the same ordering key are delivered in publish order within a subscription. Different keys may be processed concurrently, and duplicates can still occur.
Pull delivery: The subscriber controls polling and concurrency and should acknowledge or negatively acknowledge each message.
Push delivery: Pub/Sub retries unsuccessful requests with exponential backoff until the delivery-attempt limit is reached. Authenticated push can use OIDC tokens to authenticate the Pub/Sub service to the endpoint.
Payload limit: A message can be up to 10 MB. Larger data should normally be stored in Cloud Storage and referenced from the message.
Service Comparisons
Delivery pattern
Google Cloud
AWS
Azure
Event fan-out
Pub/Sub topic and subscriptions
SNS topics with SQS, Lambda, or HTTPS consumers
Event Grid topics
Queue-style decoupling
Pub/Sub subscription with competing consumers
SQS standard or FIFO queues
Service Bus queues
High-throughput streaming
Pub/Sub
Kinesis Data Streams or MSK
Event Hubs
Serverless consumption
Cloud Run, Cloud Functions, or GKE
Lambda, ECS, or EKS
Functions, Container Apps, or AKS
Pub/Sub is particularly suitable when one event must reach several independent consumers or when producers and consumers operate on different schedules.
Security and Operations
Grant narrowly scoped predefined roles such as roles/pubsub.publisher and roles/pubsub.subscriber, and keep publisher and subscriber identities separate.
A dead-letter topic requires the Pub/Sub service agent to have permission to publish to it, commonly through roles/pubsub.publisher.
Use authenticated push with narrowly scoped audiences, and validate incoming webhook requests at the receiving service.
Pub/Sub encrypts data in transit and at rest by default; use customer-managed encryption keys when organizational policy requires them.
Monitor pubsub/subscription/num_undelivered_messages, pubsub/subscription/oldest_unacked_message_age, and pubsub/subscription/num_push_requests_dropped. Instrument publisher-side errors separately and use logs for endpoint failures.
Review the topic’s message-storage policy, regional requirements, quotas, and expected traffic spikes. Pub/Sub scales automatically, but publishers and subscribers can still encounter throttling under extreme load.
Production Best Practices
Acknowledge only after the required durable processing has succeeded, and use lease management for long-running handlers.
Make handlers idempotent because redelivery and duplicate processing are normal operational realities.
Configure bounded retries and dead-letter handling for poison messages; inspect and validate dead-letter messages before replaying them.
Version event schemas, publish schema versions as attributes, and reject incompatible payloads without crashing the entire consumer.
Keep payloads small and store larger objects in Cloud Storage.
Monitor backlog age, delivery failures, and dead-letter volume rather than throughput alone.
Test dependency outages, expired acknowledgements, malformed messages, throttling, and dead-letter replay procedures before production incidents occur.
38
What is the difference between push and pull Pub/Sub subscriptions?
KnownRevise
A push subscription lets Pub/Sub deliver messages to an HTTPS endpoint, while a pull subscription requires the subscriber to retrieve messages from Pub/Sub. By default, both provide at-least-once delivery, so subscribers must tolerate duplicates and acknowledge messages only after successful processing.
Read full answerHide full answer
Applies toGoogle Cloud Pub/Sub (all current versions)
How Delivery Works
With a push subscription:
Pub/Sub sends an HTTPS POST request to the configured push endpoint.
The request contains the message data, attributes, and delivery metadata.
The endpoint processes the message and returns a successful HTTP status.
Pub/Sub treats a 2xx response as an acknowledgment; failed or timed-out requests are retried.
With a pull subscription:
The subscriber connects to Pub/Sub using the Pull or StreamingPull API.
Pub/Sub returns available messages and assigns them an acknowledgment deadline.
The subscriber processes each message and explicitly acknowledges it.
Unacknowledged messages become eligible for redelivery when their lease expires.
Push Versus Pull
Area
Push subscription
Pull subscription
Delivery initiation
Pub/Sub initiates the request
Subscriber initiates the request
Typical latency
Low latency without waiting for the subscriber to poll
StreamingPull provides low latency; synchronous Pull depends on client behavior
Scale control
Pub/Sub initiates and retries requests; the endpoint must scale for received and redelivered traffic
The subscriber controls concurrency and worker count, subject to subscription and project quotas
Network exposure
Requires a reachable HTTPS endpoint, usually protected with OIDC authentication
Requires outbound access to Pub/Sub; no public application endpoint is needed
Processing control
The endpoint controls acknowledgment through its HTTP response
The client controls acknowledgment and lease extension
Infrastructure
Can use Cloud Run, GKE, or another HTTPS service
Usually uses continuously running workers on GKE, Compute Engine, or Cloud Run
Cost profile
Can reduce idle compute with serverless scaling, but delivery attempts and endpoint traffic still cost money
Keeps subscriber infrastructure running, even when the backlog is empty
Delivery Semantics and Failure Modes
At-least-once delivery by default: Messages can be delivered more than once. A duplicate can occur when processing succeeds but the acknowledgment is lost, or when a subscriber fails after performing side effects.
Exactly-once delivery: Pull subscriptions can enable Pub/Sub exactly-once delivery, which prevents duplicate delivery by the service during the subscription’s message-retention period. It does not make arbitrary external side effects exactly-once, so handlers should remain idempotent.
Acknowledgment deadline: Both subscription types use an acknowledgment deadline. Pull subscribers can extend it with ModifyAckDeadline; push endpoints must return 2xx, and non-success responses or timeouts cause redelivery under the retry policy.
Redelivery: Repeated processing failures increase redelivery attempts. Configure a dead-letter topic when messages repeatedly fail and need separate investigation.
Ordering: When ordering is enabled, messages published with the same ordering key are delivered in order. Duplicate or retried messages must still be handled idempotently.
External side effects: A database transaction can make a database write idempotent, but it does not automatically make an entire external workflow exactly-once.
Security and Operational Trade-offs
Push security: Use authenticated push with an OIDC token. The receiving service must validate the token’s signature, issuer, audience, and expected service account. Do not expose an unauthenticated public endpoint or treat URL secrecy as authentication.
Pull security: Grant the workload’s identity the roles/pubsub.subscriber role. Workload Identity on GKE or attached service identities are preferable to downloaded service-account keys.
Backpressure: A slow push endpoint can accumulate retries, while a slow pull consumer can accumulate an unacknowledged backlog. Set retention, dead-letter, and alerting policies based on business recovery requirements.
Capacity planning: Test autoscaling and Pub/Sub quotas with realistic burst traffic rather than steady-state averages.
Best Practices
Make handlers idempotent, using the Pub/Sub message ID or an application-level deduplication key.
Acknowledge only after the message has been durably processed.
Extend the acknowledgment deadline for long-running work.
Return a 2xx response only when the push endpoint considers processing successful.
Use separate dead-letter topics and alerts for poison messages.
Choose push for serverless endpoints and naturally bursty workloads; choose pull when the subscriber needs explicit concurrency control, private networking, or long-lived processing.
39
What delivery and ordering guarantees does Pub/Sub provide?
KnownRevise
Google Cloud Pub/Sub provides at-least-once delivery by default and offers best-effort ordering within an ordering key for subscriptions with message ordering enabled. Exactly-once delivery is also available for pull subscribers, but it does not replace idempotent business logic or provide a global ordering guarantee.
Read full answerHide full answer
Applies toGoogle Cloud Pub/Sub (current service; the guarantees described apply to all versions)
Delivery Semantics
When a subscriber receives a message, Pub/Sub starts an acknowledgement deadline. The subscriber must acknowledge the message after successfully processing it.
Not acknowledged: The acknowledgement deadline expires and Pub/Sub redelivers the message.
Duplicate delivery: Possible by default because a subscriber may finish processing but crash before sending the acknowledgement.
Message retention: Unacknowledged messages are retained according to the subscription’s message-retention configuration, subject to Pub/Sub quotas and service availability.
Exactly-once delivery can reduce redeliveries after a successful acknowledgement, but it does not guarantee exactly-once effects in an external system. For example, a subscriber could update a database and then crash before acknowledging the message. The message might be processed again, so the database update should still use an idempotency key or a transactional design.
Ordering Semantics
Pub/Sub ordering depends on an ordering key assigned by the publisher. Messages published with the same key are intended to be delivered in order to a subscription with message ordering enabled.
Important limitations include:
Ordering applies within an ordering key, not across different keys.
Messages without an ordering key have no meaningful ordering guarantee.
Ordering is regional; messages published or consumed in different regions should not be treated as globally ordered.
A subscriber must process messages for the same key sequentially. Concurrent processing can defeat application-level ordering.
If a subscriber fails or an acknowledgement deadline expires, redelivery and retries can complicate the sequence. Therefore the guarantee is best effort rather than an absolute, failure-free ordering contract.
Exactly-once delivery and ordering are independent features that can be enabled together: exactly-once is supported by pull subscribers only, and because successfully acknowledged messages are not redelivered, it reinforces rather than weakens the per-key order.
A common design is to use a customer ID, account ID, or entity ID as the ordering key. This keeps events for one entity ordered while allowing different entities to be processed in parallel.
Production Failure Modes
The most important operational failure mode is choosing the wrong acknowledgement timing:
Acknowledge too early: The message may be removed from the subscription before processing completes, causing data loss.
Acknowledge too late: The message may be delivered again, causing duplicate processing.
Poison messages: A message that repeatedly fails can block progress for its ordering key. Configure a dead-letter topic and maximum delivery attempts where appropriate.
Consumer outages: Messages accumulate in the subscription and are delivered after consumers recover, within retention and quota limits.
Incorrect scaling: Adding consumers does not mean events for the same ordering key are processed concurrently. Use key-aware partitioning or a single logical consumer per key.
Comparison With AWS and Azure
Requirement
Google Cloud Pub/Sub
AWS
Azure
Default delivery
At least once
SNS/SQS Standard at least once
Service Bus at least once
Keyed ordered processing
Ordering keys with ordering enabled
SQS FIFO MessageGroupId
Service Bus sessions
Exactly-once processing option
Available for pull subscribers; still requires idempotency
Usually implemented with deduplication or idempotent consumers
Usually implemented with idempotent or transactional processing
Choose Pub/Sub when the application needs managed publish/subscribe routing, multiple independent subscriptions, key-based serialization, and regional Google Cloud integration. Use SQS FIFO or Service Bus sessions when a platform team specifically needs their queue semantics or stronger operational integration with AWS or Azure.
Best Practices and Monitoring
Enable ordering per subscription with gcloud pubsub subscriptions create --enable-message-ordering, and publish keyed messages with gcloud pubsub topics publish --ordering-key.
Use idempotent handlers even when exactly-once delivery is enabled.
Acknowledge only after the business transaction has committed.
Alert on sustained backlog growth or increasing message age rather than only on message count.
Grant publishers only permission to publish, and grant subscribers only the permissions required to consume and acknowledge.
Use a dead-letter topic for poison-message handling, and test recovery procedures before production traffic reaches the failure path.
40
How does Google Cloud Dataflow relate to Apache Beam?
KnownRevise
Google Cloud Dataflow is Google Cloud’s managed service for running large-scale batch and streaming data-processing pipelines. Apache Beam is the open-source, multi-language SDK and programming model used to define those pipelines; Dataflow is one of Beam’s execution runners.
The relationship is similar to using portable SQL with different execution engines:
Apache Beam defines transformation logic using constructs such as PCollection, transforms, windows, triggers, and side inputs.
Beam runner executes the pipeline on a processing platform.
Dataflow Runner is Google Cloud’s implementation of that runner.
The same Beam pipeline can potentially run on other runners, including Flink, Spark, or the local runner, although production features and operational behavior differ.
Beam supports Java, Python, and Go. A typical production pipeline therefore consists of Beam SDK code, runner-specific options, and deployment through gcloud or CI/CD.
How Dataflow Executes a Pipeline
The launcher submits the Beam pipeline and its options to Dataflow.
Beam constructs a logical processing graph from the SDK operations.
Dataflow Runner translates that graph into an executable Dataflow job.
Dataflow provisions workers, distributes work, checkpoints streaming state, and restarts failed work when necessary.
Sources such as Pub/Sub, BigQuery, and Cloud Storage provide input; sinks such as BigQuery, Cloud Storage, and Bigtable receive output.
Dataflow supports both models:
Batch: Processes bounded input and completes the job.
Streaming: Processes unbounded input continuously, using windows, triggers, and state to aggregate events over time.
Dataflow is a managed execution service: Google provisions and operates worker compute, while customers configure pipeline code, worker capacity, and autoscaling. Costs can still arise from worker processing and connected resources such as Pub/Sub, BigQuery, and Cloud Storage usage.
Production Behavior and Failure Modes
Dataflow retries failed work and restores streaming state from checkpoints. Nevertheless, an end-to-end pipeline is not automatically exactly-once. Retries, source redelivery, and sink writes can produce duplicates, so sinks should support idempotency where possible.
Important operational concerns include:
Poison messages: Invalid records can repeatedly fail and block work. Validate records early and route persistent failures to a dead-letter topic.
Skewed work: A hot source partition or key can create stragglers that aggregate autoscaling metrics do not reveal.
Dependency outages: A temporarily unavailable sink causes backlogs, worker growth, and higher cost.
State growth: Unbounded streaming state increases checkpoint time and recovery duration.
Job failures: Monitor failed tasks, worker restarts, backlog age, throughput, and checkpoint health in Cloud Monitoring and Cloud Logging.
Use a dedicated worker service account with only the required Pub/Sub, BigQuery, and storage permissions. Avoid user credentials or downloaded service-account keys. For sensitive workloads, evaluate private connectivity, VPC Service Controls, and customer-managed encryption where supported.
Cloud Service Comparison
Cloud
Closest managed alternatives
Key distinction
Google Cloud
Dataflow
Managed Beam execution with integrated autoscaling, monitoring, and batch/streaming support
AWS
Amazon Managed Service for Apache Flink; Glue and EMR for batch
Does not provide a direct one-to-one equivalent to the Beam-managed experience
Azure
Microsoft Fabric Real-Time Analytics; Azure Synapse, Databricks, or HDInsight for other workloads
No direct one-to-one Beam-managed equivalent; Azure Stream Analytics is scheduled for retirement on 30 November 2027
Choose Dataflow when the team values a portable Beam model, unified batch and streaming development, and managed execution without operating Dataflow worker infrastructure or a Flink or Spark cluster.
Best Practices and Common Pitfalls
Test locally with the Beam Direct Runner, then validate representative scale and failure cases on Dataflow.
Make sink writes idempotent, for example by deduplicating on event_id.
Partition and cluster BigQuery output tables for the expected query patterns.
Monitor backlog age and error rate, not only worker utilization.
Keep the Beam SDK version compatible with the Dataflow runtime and test streaming updates before rollout.
Define source, sink, and dead-letter retention policies; Dataflow itself does not back up application data.
41
What does Google Cloud Dataproc provide?
KnownRevise
Google Cloud Dataproc is a managed service for running Apache Spark and Hadoop workloads. It provides autoscaling Compute Engine-based clusters and Dataproc Serverless for managed Spark batch processing and interactive sessions, with integration into Google Cloud storage, IAM, networking, logging, and monitoring.
Dataproc abstracts much of the operational work required to run Spark and Hadoop:
Cluster mode: Dataproc creates Compute Engine instances in your VPC. A master node coordinates the workload, while worker nodes run Spark executors or other processing tasks.
Serverless mode: Dataproc Serverless runs Spark workloads without requiring users to configure or manage persistent virtual machines. Interactive Spark sessions are available in supported regions and runtimes.
Storage: Jobs commonly read and write Cloud Storage data. Cluster images can also use Hadoop Distributed File System and other connectors.
Networking: Cluster nodes communicate through the VPC network and can use only private IP addresses.
Lifecycle: Batch clusters are normally disposable. Teams recreate them from versioned configuration rather than maintaining long-lived compute for every workload.
Spark still executes an application driver, partitions input into tasks, and schedules those tasks on executors. Dataproc manages the compute and supporting services, but application-level concerns such as data skew, memory pressure, retries, and output correctness remain the job developer’s responsibility.
Core Capabilities
Apache Spark: Supports Spark SQL, PySpark, Scala and Java applications, Structured Streaming, MLlib, and custom Spark processing.
Hadoop ecosystem: Cluster images can provide components such as HDFS and Hive, depending on the selected image version and optional components.
Autoscaling: Cluster-mode workers can scale between configured minimum and maximum worker counts.
Job submission: Workloads can be submitted through the gcloud CLI and the Dataproc job or batch APIs.
Configuration: Clusters support custom machine types, image versions, optional components, labels, networking, encryption, and autoscaling.
Service integration: Dataproc works with Cloud Storage, BigQuery, Cloud Monitoring, Cloud Logging, Artifact Registry, IAM, and VPC networking.
Production Operations
Dataproc is designed primarily for ephemeral, data-intensive batch processing.
Monitoring: Use Cloud Monitoring for cluster and job metrics and Cloud Logging for driver, executor, and system logs. Consistent labels support dashboards, incident analysis, and charge attribution.
Failure recovery: Failed executors can normally be replaced, but retrying a failed job can repeat partially completed work. Make writes idempotent and use checkpointing where appropriate.
State: Do not treat local cluster disks as durable storage. Keep code and configuration in source control, dependencies in versioned artifact repositories, and input and output data in Cloud Storage or another durable service.
Upgrades: Test new image versions against representative jobs. For critical pipelines, validate results on a canary cluster before replacing production clusters.
Capacity: Compute Engine quotas, subnet IP capacity, and zonal resource availability can delay cluster creation or autoscaling.
Cleanup: Use external automation such as Cloud Scheduler and Workflows to delete disposable clusters after successful pipeline completion.
Dataproc does not itself guarantee exactly-once processing. End-to-end correctness depends on Spark retries, checkpointing, and the semantics of the destination system.
Security and Cost
Run cluster nodes in a private subnet without public IP addresses. Enable Private Google Access for required Google APIs and use Cloud NAT when general outbound internet access is necessary; restrict permitted destinations with network controls where possible.
Grant users and service accounts only the IAM permissions needed to submit jobs and access data.
Use VPC Service Controls when required by data-protection policy.
Keep Cloud Storage buckets private and use narrowly scoped IAM access.
Use Google Cloud’s default encryption or customer-managed encryption keys when required.
Use Cloud Audit Logs for administrative activity and enable Data Access logs where the required audit coverage includes storage access.
Dataproc pricing includes cluster-management charges, Compute Engine usage, persistent disks, networking, and storage. Configured minimum workers continue incurring charges while a cluster exists. Serverless removes idle cluster costs, although BigQuery may be less expensive for SQL-only analytics.
Choosing the Right Service
Requirement
Preferred GCP service
Reason
Spark and Hadoop batch processing
Dataproc
Managed Spark and Hadoop execution with cluster or Serverless modes
Managed SQL warehouse analytics
BigQuery
Serverless, pay-per-query analytics without cluster management
Apache Beam batch and streaming pipelines
Dataflow
Managed execution for Beam pipelines
Custom containerized workloads on Kubernetes
GKE
Kubernetes scheduling and broader deployment flexibility
Interactive Spark development
Dataproc Serverless or cluster mode
Serverless interactive sessions where offered; cluster mode provides a dedicated environment with direct node control
The closest AWS equivalents are Amazon EMR and EMR Serverless. Current Azure managed Spark counterparts include Azure Databricks, Microsoft Fabric Spark, and Synapse Spark; Azure HDInsight is being retired and is not appropriate for new deployments. Dataproc is especially attractive when the workload already uses Cloud Storage, BigQuery, and Google Cloud IAM.
Best Practices
Use version-controlled Terraform for repeatable cluster definitions.
Pin image and runtime versions, then test upgrades before production rollout.
Set autoscaling limits from measured workload requirements.
Configure alerts for failed jobs, executor failures, resource exhaustion, and abnormal cluster growth.
Separate development, staging, and production through projects, networks, IAM policies, and labels.
Delete disposable clusters automatically after successful pipeline completion.
42
What is Google Cloud Composer?
KnownRevise
Google Cloud Composer is Google Cloud’s managed service for Apache Airflow, used to define, schedule, and operate data-engineering workflows as DAGs. Google manages the Airflow runtime and supporting infrastructure, while teams manage DAGs, dependencies, access, and operational policy.
Read full answerHide full answer
Applies toGoogle Cloud Composer 2 and 3 with Apache Airflow 2.10.5 or Airflow 3.x image versions
How It Works
A Composer environment runs managed Airflow infrastructure on Google Kubernetes Engine, with the Airflow metadata database in Cloud SQL. Google operates the control plane, Kubernetes infrastructure, database service, and maintenance.
DAG source is stored in an associated Cloud Storage bucket and made available to Airflow for parsing.
The Airflow scheduler evaluates schedules and dependencies and creates task instances.
Worker pods run operators that execute the actual work.
Task state is stored in the Airflow metadata database, logs are written to Cloud Logging, and operational metrics are available in Cloud Monitoring.
Composer orchestrates work in other Google Cloud services through operators. For example, a DAG can query BigQuery, launch a Cloud Run Job, create a GKE workload, publish to Pub/Sub, or submit a Dataflow job.
Core Capabilities
Scheduling: Cron, date, interval, and timedelta schedules are supported. Asset, event, and calendar-trigger availability depends on the Composer generation and Airflow image version. UTC is the default, but DAGs may define another timezone; catchup controls whether missed scheduled intervals are backfilled.
Dependency management: Tasks form a directed graph and wait for their upstream dependencies to succeed.
Retries and timeouts: Operators can define retry limits, delays, and execution timeouts.
Dynamic task mapping: Airflow 2.3 and later can expand one task into multiple task instances from a runtime-sized collection.
Custom logic: Python tasks and supported third-party operators cover workflows that cannot be expressed with simple service calls.
Environment isolation: Each environment has its own configuration, Airflow metadata database, and DAG bucket. It also has an assigned service account, which can be shared but should normally be dedicated and minimally privileged.
Production Operations
Use gcloud to inspect environments:
gcloud composer environments list --project=example-project --location=us-central1 lists environments.
gcloud composer environments create example-environment --project=example-project --location=us-central1 --environment-size=ENVIRONMENT_SIZE_SMALL creates an environment using the default image.
Monitor scheduler delay, task duration, failures, worker saturation, Cloud SQL load, DAG parsing errors, and service quotas. Configure Cloud Logging sinks and retention explicitly where compliance or incident investigation requires it.
Composer does not back up application data processed by tasks. Keep DAG source and dependency manifests in version control, use Cloud Storage versioning or retention controls where appropriate, and use the backup and restore facilities supported by the selected Composer generation.
Image-version upgrades require compatibility testing because Airflow upgrades can change operator behavior, deprecated parameters, dependency resolution, and task semantics. Test DAGs in a non-production environment and verify restoration procedures before production use.
Failure Modes and Security
Common incidents include:
Scheduler delay: metadata-database pressure or expensive DAG parsing delays task creation.
External-service failures: quotas, authentication errors, network restrictions, and service outages fail otherwise valid tasks.
Dependency failures: an unsuccessful upstream task prevents its downstream chain from progressing.
DAG deployment errors: syntax errors, missing packages, and import-time exceptions prevent new runs.
Run tasks under a dedicated service account with only the permissions required by its operators. Prefer attached service-account identities over downloaded keys, use Workload Identity Federation where the target supports it, and restrict Airflow UI and operator access with IAM. For sensitive workloads, evaluate private networking, VPC Service Controls, and customer-managed encryption keys where supported.
AWS and Azure Comparison
Capability
Google Cloud Composer
Amazon Managed Workflows for Apache Airflow (MWAA)
Azure Data Factory
Core model
Managed Apache Airflow
Managed Apache Airflow
Managed data-integration workflows
Workflow language
Python DAGs
Python DAGs
Pipelines and activities, not Airflow DAGs
Best fit
GCP-native, code-driven orchestration
AWS-native Airflow workflows
Data ingestion, transformation, and integration
Key distinction
Managed GKE-based Airflow runtime
Closest direct Airflow equivalent
Not an Apache Airflow service
Choose Composer for flexible Python-based orchestration across GCP services. For simple scheduled jobs, Cloud Scheduler with Cloud Run or Workflows may be simpler. Composer is less suitable for sub-second event processing, continuously running streaming applications, or workflows dominated by simple API calls where Airflow’s scheduling overhead is unnecessary.
43
What is a GCP VPC and how is it global?
KnownRevise
A Virtual Private Cloud (VPC) network is a logically isolated private network in Google Cloud. It is a global resource that can contain regional subnets across multiple regions and zones, allowing supported workloads in different locations to communicate privately within one network boundary.
Read full answerHide full answer
Applies toGoogle Cloud VPC networking (all current versions)
Core Model
VPC network: A software-defined network managed by Google that defines IP address space, routing, and private connectivity. It is not a physical network or a single data center.
Resource hierarchy: A VPC belongs to a project, which can be placed beneath folders and an organization. With Shared VPC, a host project owns the network while service projects attach supported workloads.
Subnet: Allocates an IPv4 range for Compute Engine instances and other supported resources. A subnet is regional and can provide secondary ranges for services such as GKE.
IAM and firewall separation: IAM controls who can administer network resources, while VPC firewall rules control which network traffic is permitted. Production environments need both.
Internal communication: Resources connected to the same VPC can communicate using internal IP addresses, subject to routing, firewall, and product-specific configuration requirements.
Global and Regional Scope
Resource
Scope
Meaning
VPC network
Global
One network can support workloads across multiple regions and zones
Subnet
Regional
Allocates IP ranges within one Google Cloud region
Compute Engine instance
Zonal
Uses an interface in a subnet in the instance’s region
Global external IPv4 address
Global
Attaches only to resources that support global external addresses
Regional external IPv4 address
Regional
Attaches only to supported resources in its region
VPC firewall rule
VPC-wide by default
Applies across the network; narrow it with targets such as tags or service accounts
Internal/external load balancer
Regional or global
Depends on the specific load-balancing product
Cloud NAT
Regional
Provides outbound internet connectivity for resources in its region
“Global” describes the VPC’s logical control and addressing scope; it does not eliminate regional latency, capacity limits, or failures. A multi-region deployment still requires regional subnets, regional dependencies, regional capacity, and per-location monitoring.
Connectivity, Routing, and Security
Google manages the VPC as a software-defined network. Its network definition and routing configuration are global resources, while packet forwarding remains tied to the regional infrastructure serving each workload.
Internet egress: Instances without external IP addresses generally need Cloud NAT for outbound internet connectivity. Cloud NAT does not accept unsolicited inbound connections.
Private access to Google services: Private Google Access sends traffic for supported Google APIs and services through private addresses.
Managed private connectivity: Private Service Connect and Private Service Access support private access to managed services through different architectural patterns.
Hybrid connectivity: Cloud VPN and Cloud Interconnect connect a VPC to on-premises environments; another cloud can be reached through VPN, peering, or an Interconnect service provider.
Default firewall behavior: VPCs include implied rules that deny incoming traffic and permit outgoing traffic. Explicit rules should restrict sensitive ports and administrative access.
Routing: Custom routes participate in route selection alongside system-generated and default routes. The most specific matching prefix is selected, with route attributes determining selection when prefixes tie. The default internet route is only the fallback.
Comparison With AWS and Azure
Capability
Google Cloud
AWS
Azure
Primary network resource
VPC network
VPC
Virtual network
Default network scope
Global
Regional
Regional
Subnet scope
Regional
Availability Zone
Virtual network, with zone-aware deployment
Static public IP scope
Global or regional
Regional
Regional
Managed private access
Private Service Connect, Private Service Access, and Private Google Access for supported Google APIs
VPC endpoints and AWS PrivateLink
Private endpoints and Azure Private Link
GCP’s global VPC simplifies multi-region addressing, but AWS and Azure can also connect regional networks. AWS commonly uses Transit Gateway; Azure commonly uses Virtual WAN or hub-and-spoke VNet peering.
Production Practices and Failure Modes
Plan address space: Use non-overlapping CIDR ranges across VPC peering, VPN, Interconnect, and Shared VPC relationships.
Monitor consumption: Track subnet address usage and relevant quotas before deploying large GKE clusters or autoscaling groups.
Apply least privilege: Lower firewall-rule priorities are evaluated first. Restrict SSH, RDP, database, and management ports to trusted sources.
Design for regional failures: Cloud NAT, subnet capacity, and regional load balancers remain regional even when the VPC is global.
Automate and audit changes: Manage networking through reviewed Terraform or configuration pipelines, enable supported firewall logging, and detect configuration drift.
Watch common failures: IP exhaustion, subnet quota limits, asymmetric routes, missing NAT egress, incorrect firewall priorities, and unavailable regional capacity are more likely than failure of the global VPC definition itself.
44
How do VPC subnets and firewall rules work in GCP?
KnownRevise
VPC subnets define regional IP address space in a Google Cloud VPC network, while firewall rules are VPC-wide, stateful controls that allow or deny traffic to and from Compute Engine instances, including the instances backing GKE nodes and managed instance groups.
VPC network: A global, private logical network owned by a project or a Shared VPC host project. Resources communicate through internal IP addresses.
Subnet: A regional IP range within a VPC network. Its address space is available to resources in every availability zone of that region, unlike an AWS subnet, which belongs to one Availability Zone.
Instance interface: Each Compute Engine network interface is attached to a subnet in the same VPC network.
Secondary IP ranges: Additional ranges configured on a subnet for alias IP addresses, commonly used for GKE pod address space. GKE can use another secondary range for Service address space.
Private Service Connect addresses: Allocated from internal address ranges reserved for Private Service Connect rather than from a subnet’s primary range.
Firewall rule: A global resource within a VPC that filters ingress or egress traffic by protocol, port, source or destination, and target.
Network tags: Labels attached to instances or interfaces. They identify workloads in firewall rules without hard-coding IP addresses.
Resource hierarchy: VPCs belong to projects, which can be organized under folders and an organization for centralized IAM and governance.
How Traffic Is Evaluated
Firewall rules are evaluated against the relevant Compute Engine virtual network interface before traffic reaches the instance.
Rules have priorities from 0 to 65535; a lower number is evaluated first.
The first matching rule determines whether traffic is allowed or denied. If no custom rule matches, the implied rules apply.
An ingress rule can match source ranges, source tags, source service accounts, protocols and ports, and target tags or service accounts.
An egress rule can match destination ranges, destination service accounts, protocols and ports, and the source workload through target tags or service accounts.
Processing is stateful: reply traffic for an allowed connection is automatically permitted.
The implied rules deny all ingress and allow all egress. Custom allow rules therefore do not restrict egress unless a higher-priority deny rule is configured.
Assigning a public IP address does not bypass firewall rules; the applicable traffic must still be allowed.
Auto-mode VPC networks create default SSH and RDP rules from 0.0.0.0/0. Custom-mode networks avoid these legacy defaults and are generally preferable for production networks.
Production Design
Use a custom-mode VPC network and create only the subnets required by each region.
Plan address space carefully. Primary and secondary ranges cannot overlap within one VPC. Primary subnet ranges can be expanded but not reduced.
Use a controlled administrative bastion reached through IAP TCP forwarding rather than exposing port 22 globally. For an IAP-forwarded bastion, allow port 22 from 35.235.240.0/20 and enforce strong IAM or OS Login controls.
Use Cloud NAT when private instances require outbound internet access without public IP addresses. Apply explicit egress firewall policy and monitor NAT port exhaustion.
Enable VPC Flow Logs and firewall-rule logging. VPC Flow Logs and firewall logs are separate signals; use firewall log entries in Cloud Logging to investigate denied traffic and create log-based alerts for unexpected denials.
Use Cloud Armor on supported HTTP(S) proxy and load-balancer frontends for layer-7 protection. VPC firewall rules remain primarily network- and transport-layer controls.
Use Shared VPC when service projects need centrally administered networking. Use VPC peering or Private Service Connect when isolated networks require controlled private reachability; neither extends firewall-policy ownership automatically.
Cloud Comparison
Capability
Google Cloud
AWS
Azure
Network scope
Global VPC with regional subnets
Global VPC with Availability Zone-specific subnets
Regional VNet with subnets
Subnet reach
Spans all zones in a region
One Availability Zone per subnet
Can span availability zones
Network filtering
Stateful firewall rules with allow and deny
Stateful security groups and stateless network ACLs
Stateful Network Security Groups and Azure Firewall
Rule-targeting dimensions
Tags and instance service accounts
Security groups and tags
Security-group membership and service tags
Common Pitfalls
Opening administrative ports to 0.0.0.0/0 exposes workloads directly to the internet.
Overlapping or poorly partitioned ranges cause routing and migration problems, while conflicting firewall priorities make policy difficult to reason about.
Firewall rules do not provide application authentication, TLS termination, rate limiting, or HTTP request filtering.
GKE firewall rules act on node interfaces rather than directly on pod interfaces; use Kubernetes NetworkPolicy for pod-level controls.
Cloud Run and Cloud Functions are not directly targeted by VPC firewall rules in the same way as Compute Engine instances; use their ingress, IAM, and VPC connectivity controls.
Source and target tags identify eligible instances within the same VPC network, not arbitrary external traffic.
45
What is Shared VPC in Google Cloud?
KnownRevise
Shared VPC is a Google Cloud networking model in which an organization’s host project owns the VPC network and subnets, while attached service projects run their own compute resources. This centralizes network configuration without moving workloads such as virtual machines into the host project.
Read full answerHide full answer
Applies toGoogle Cloud Shared VPC (current)
How It Works
Host project: Owns the VPC network, subnets, routes, firewall rules, Cloud Router, and other shared network resources.
Service projects: Remain responsible for their own Compute Engine instances, managed instance groups, Cloud Run services, GKE nodes, and other resources.
Shared subnets: Are defined once in the host project and selected when creating resources in an attached service project.
Resource ownership: A virtual machine remains in its service project, including its IAM policies, logs, quotas, and billing, even though its network interface uses a subnet owned by the host project.
Organization requirement: The host and service projects generally belong to the same Google Cloud organization. An organization administrator designates the host project and controls which projects can participate.
Shared VPC centralizes network addressing and policy, but it does not share virtual machines, disks, databases, or other compute resources.
IAM and Security
IAM combines permissions from both projects:
Service project permissions: Determine who can create resources in that project.
Host project subnet permissions: Determine who can attach resources to a shared subnet.
Network administration: Host project network administrators manage central routes, firewall rules, and subnets.
Least privilege: Grant roles/compute.networkUser on only the required shared subnet rather than granting host-project network administration.
The host project’s firewall rules and routes affect every attached service project. A bad route, overly permissive ingress rule, or exhausted shared IP address can therefore create a multi-project incident.
Shared VPC Compared with Other Models
Model
Ownership
Multi-project use
Closest external equivalent
Google Cloud Shared VPC
One host project owns the network
Service projects use shared subnets
AWS VPC sharing
Google Cloud VPC peering
Each side owns a separate VPC
Peer-to-peer connection between two VPCs
AWS VPC peering
Azure hub-and-spoke
A hub VNet owns central connectivity
Spoke VNets peer with the hub
Azure hub-and-spoke
AWS Transit Gateway
A gateway connects multiple VPCs
Central transit without sharing subnets
Google Cloud VPC peering plus Cloud Router
Choose Shared VPC when multiple projects need centrally managed subnets and consistent network controls. Use VPC peering when separate VPC ownership is more important than sharing a common address space.
Production and Operational Behavior
Address management: Allocate non-overlapping CIDR ranges centrally, such as 10.10.0.0/16, and divide them into regional subnets.
Connectivity: Use Cloud Router and dynamic routes for on-premises or partner connectivity. Shared VPC does not itself create a VPN or Cloud Interconnect connection.
Egress: A centrally managed Cloud NAT can provide outbound connectivity, but its configuration and address capacity affect all attached workloads that use that NAT and subnet.
Monitoring: Monitor host-project firewall metrics, router health, NAT utilization, and subnet address exhaustion. Service-project dashboards should correlate Compute Engine failures with host-network health.
Logging: Enable Firewall Rules Logging and retain audit logs according to the organization’s incident and compliance requirements.
Quotas and billing: Network configuration is administered centrally, while compute resource quotas remain associated with the owning service project. Network charges may be attributed to the host project.
Failure isolation: Centralization improves consistency but increases blast radius. Test route and firewall changes against representative service projects before broad rollout.
Best Practices and Common Pitfalls
Manage the host project with Terraform: Treat host-project VPCs, subnets, routes, firewall rules, and IAM bindings as reviewed infrastructure code.
Use separate subnets by environment and region: For example, use 10.10.10.0/24 for production and 10.20.10.0/24 for staging.
Do not grant broad host-project roles: Use roles/compute.networkUser on individual subnets wherever possible.
Plan capacity centrally: Monitor consumed IP addresses and Cloud NAT ports before expansion.
Control project attachment: Review organization policy and audit logs to prevent unauthorized service projects from joining.
Test detach and recovery procedures: Attached resources and subnet dependencies can prevent service-project detachment or host-project deletion.
Avoid overlapping CIDR ranges: Shared VPC does not make overlapping subnet ranges routable.
46
VPC Peering vs Shared VPC in Google Cloud
KnownRevise
VPC peering creates a private, direct connection between two independent VPC networks, while Shared VPC allows multiple projects to use one centrally managed VPC network. Peering is a connectivity relationship between networks; Shared VPC is an ownership and IAM model for sharing a network across projects.
Read full answerHide full answer
Applies toGoogle Cloud VPC networking and Shared VPC (current stable)
Core Difference
Aspect
VPC peering
Shared VPC
Main purpose
Connect two separate VPC networks
Let multiple projects use one VPC network
Network ownership
Each project owns its VPC
The host project owns the VPC and subnets
Administrative model
Each side manages its own network
The host project centrally controls networking
Connectivity
Private-IP communication between the peered networks
Service-project resources use the host project’s network directly
Transitivity
Non-transitive
Not a peering model; resources are already on the shared network
Typical use case
Connecting isolated networks or environments
Sharing a platform network across an organization
A common misconception is that these features are alternatives. They solve different problems and can be combined: service projects can use a Shared VPC network, and that shared network can peer with another VPC.
How VPC Peering Works
A peering connects two VPC networks directly, either within one project or across projects. Google Cloud exchanges routes between the peered networks automatically, so instances with compatible internal IP addresses can communicate over private IP.
Important constraints include:
Non-transitivity: If VPC A peers with VPC B and VPC B peers with VPC C, VPC A cannot reach VPC C through those peerings.
No overlapping subnet ranges: The peered networks must not use the same IP address space.
Separate firewall policies: Each VPC controls its own firewall rules. Each side must allow the initial ingress traffic, while stateful firewall behavior permits return traffic for an established connection without a separate return rule. Independent traffic initiated from the other direction requires its own ingress rule.
No central network administrator: Peering does not transfer ownership of either network.
Private connectivity: Peering provides internal-network connectivity; it is not an internet gateway or an IAM-sharing mechanism.
Peering is useful when teams or environments need controlled, explicit links without giving one project administrative ownership of the other network.
How Shared VPC Works
With Shared VPC, a host project owns the VPC network, subnets, routes, and firewall configuration. One or more service projects attach to the host and create resources using its shared subnets.
For example, a company might have:
A production host project owning shared subnets such as 10.20.0.0/16.
A payments service project running Compute Engine instances in 10.20.10.0/24.
A billing service project running GKE clusters whose nodes use 10.20.20.0/24.
There is no additional VPC peering between the host and service projects because their resources already use the same VPC network.
Shared VPC is useful for enforcing central network controls while allowing product teams to manage their own compute, Kubernetes, and application resources. Attachment requires appropriate IAM permissions and may be constrained by organization policies.
IAM and Security
Peering and Shared VPC both work alongside IAM, but they enforce access differently:
Peering: IAM permissions remain project-specific. Network reachability does not automatically grant access to VMs, disks, APIs, or data.
Shared VPC: Service-project principals need an appropriate role, commonly roles/compute.networkUser, on each shared subnet they use. roles/compute.networkAdmin provides broader administrative authority.
Least privilege: Grant service teams access only to the shared subnets and resources they require.
Auditability: Review host-project IAM changes and firewall changes centrally, while retaining service-level resource logs and ownership.
Shared VPC introduces a larger blast radius because one team controls connectivity for several service projects. Apply change management, infrastructure-as-code approvals, and monitoring to host-project changes.
Operational Trade-offs
Peering failures: Common causes include overlapping IP ranges, missing firewall rules, incorrect peering configuration, route or quota problems, and incorrect assumptions about transitive routing.
Shared VPC failures: Common causes include incomplete service-project attachment, missing roles/compute.networkUser, shared-subnet address exhaustion, host-project networking limits, or an organization policy preventing the attachment.
Monitoring: Monitor reachability, firewall-rule logs, relevant service SLIs, and peering state. For Shared VPC, monitor shared-subnet capacity and host-project networking because subnet exhaustion or a bad centralized change can affect every attached project.
Recovery: Keep network definitions in Terraform or another declarative system, and validate firewall, IAM, and subnet changes before applying them.
Backups: Back up network configuration and relevant IAM policies, but do not treat a configuration snapshot as a substitute for rebuilding network resources from source control.
Choose VPC peering when independent networks need a direct bilateral connection. Choose Shared VPC when multiple projects should operate on one centrally governed network.
47
What is Google Cloud NAT?
KnownRevise
Google Cloud NAT is a managed, regional service that provides outbound internet connectivity to Compute Engine instances that do not have external IP addresses. It translates private source addresses and ports to a shared public NAT IP address without providing inbound connectivity.
A Cloud NAT gateway is created on a Cloud Router and serves selected subnets in one VPC network and region.
A VM without an external IP initiates an outbound connection.
The VPC route directs eligible traffic through the Cloud NAT gateway.
Cloud NAT translates the VM’s private IP and source port into a public NAT IP and allocated port.
The external service sees the Cloud NAT IP, not the VM’s private IP.
Return traffic is translated back to the originating VM.
Cloud NAT supports outbound TCP, UDP, and ICMP traffic. It supports both manually configured static NAT IP addresses and automatic allocation of ephemeral NAT IP addresses as demand grows.
Private VM egress: Instances can access internet destinations without exposing themselves through public IP addresses.
Regional scope: One gateway can serve multiple subnets in the same region; resources in other regions require their own gateway or another egress design.
Outbound only: Cloud NAT cannot accept unsolicited inbound connections, perform port forwarding, or replace an external application load balancer.
Traffic forwarding: It supports outbound connectivity to internet destinations and, where routing permits, peered VPC networks.
Key Components and Design Choices
Cloud Router: Hosts the Cloud NAT configuration and participates in VPC route management.
Subnet selection: The gateway can serve specific subnets or all subnets in the region.
NAT IP addresses: Manual mode gives predictable public egress addresses for firewall allowlists; automatic mode simplifies capacity management.
IP version: Cloud NAT provides IPv4 egress. IPv6-only architectures need a separate IPv6 networking design.
Logging: Optional Cloud NAT logs provide visibility into translated flows, including source VM, destination, protocol, port, and traffic volume.
Production Operations
Cloud NAT is commonly used when workloads need controlled egress without directly exposing individual VMs.
Monitoring: Track connection counts, port utilization, allocation failures, processed bytes, and NAT IP consumption in Cloud Monitoring.
Logging: Enable Cloud NAT logging during validation and incidents, then evaluate the additional log volume and cost for production.
Capacity: Investigate high port utilization and allocation failures, especially with many short-lived connections or substantial UDP traffic.
Quotas: Monitor project and regional quotas for Cloud NAT gateways, NAT IP addresses, and related networking resources.
Backups and upgrades: There are no customer-managed gateway instances, snapshots, or software upgrades. Google operates the data plane, but the surrounding routing, IAM, firewall, and logging configuration remains your responsibility.
Incident diagnosis: Check whether the subnet route points to the correct Cloud Router, whether the gateway has available NAT IP capacity, and whether VPC firewall rules permit the required traffic.
Useful commands include:
gcloud compute routers nats list --router=example-router --region=us-central1 --project=example-project
Cloud NAT centralizes outbound public addresses, making provider-side IP allowlisting practical. However, it is an address-translation service, not an application-layer firewall: it does not inspect URLs, payloads, or application behavior. Use VPC firewall policies, DNS controls, secure web proxies, and workload-aware egress controls when stronger policy enforcement is required.
The main charges are for allocated NAT IP addresses and processed data, in addition to any applicable network, logging, and observability charges. Manual mode can improve address predictability but requires capacity planning; automatic mode reduces allocation work but may make the exact public address less predictable unless carefully configured.
AWS and Azure Equivalents
Cloud
Equivalent
Scope and notable behavior
Google Cloud
Cloud NAT
Regional managed egress for private VMs; no inbound access
AWS
NAT Gateway
Zonal resource; production designs generally use one per Availability Zone
Azure
NAT Gateway
Associated with subnets and backed by configured public IP resources
Choose Cloud NAT when private Compute Engine workloads need centralized outbound access. Use Cloud Load Balancing, Cloud Run, GKE, or another ingress-capable service when external users must initiate connections.
48
What is Private Google Access in Google Cloud?
KnownRevise
Private Google Access provides private connectivity from Compute Engine instances and supported hybrid workloads to Google APIs and supported Google Cloud services without sending service traffic over the public internet. It provides network routing, not authorization; IAM permissions are still required.
Read full answerHide full answer
Applies toGoogle Cloud Private Google Access (current)
How It Works
For Compute Engine, Private Google Access is enabled per subnet. Google Cloud automatically creates and manages the appropriate routes for eligible VM network interfaces, including existing instances; no manually created route is required.
The VM does not need an external IP address.
Cloud NAT is not required to reach supported Google APIs and services.
Service traffic follows Google’s internal network rather than the public internet.
DNS generally resolves the service’s public endpoint, but the Private Google Access route directs the connection over Google’s private infrastructure.
The workload still needs the appropriate IAM permissions, such as roles/storage.objectViewer when reading Cloud Storage objects.
Private Google Access does not provide general internet access. Arbitrary internet destinations still require Cloud NAT, a proxy, or another approved egress design.
Supported Workloads
Compute Engine instances: Access Cloud Storage, BigQuery, Pub/Sub, and supported Google APIs without external addresses.
GKE private clusters: Nodes and node-local workloads can reach required Google services through Private Google Access. The managed GKE control plane does not rely on the customer VPC’s Private Google Access route.
Cloud Batch and data-processing jobs: Temporary workers can retrieve inputs or publish results without public IP addresses.
Hybrid workloads: On-premises hosts connected to a VPC through HA VPN or Cloud Interconnect can access supported Google APIs over private connectivity.
Only supported Google destinations use Private Google Access. A service that instead requires a PSC endpoint, a service-owned private IP, or another interface must be connected using that mechanism.
Differences from Other Private Connectivity Options
Option
Primary purpose
Endpoint or address requirement
Outbound internet path
Private Google Access
Access Google APIs and supported services
No
No
Private Service Connect
Privately consume supported managed services
PSC endpoint
No
Private Service Access
Allocate supported services’ private IPs through VPC peering
Service-owned private IP range
No
Cloud NAT
Provide outbound internet connectivity for VMs
No
Yes
For Cloud SQL, PSC connectivity uses a PSC endpoint, while direct private-IP connectivity uses Private Service Access and VPC peering. Private Google Access is not the mechanism that allocates Cloud SQL private addresses.
Private Google Access is also distinct from VPC Service Controls. VPC Service Controls adds perimeter-based protections against unauthorized access and data exfiltration; it does not replace the routing supplied by Private Google Access.
Configuration and Validation
Private Google Access is a subnet setting; there is no network-wide switch. Enable it on each subnet that needs it:
The VM or node has no external IP address when private-only connectivity is required.
DNS resolution is working.
The workload’s service account has the required IAM roles.
Private Google Access is enabled on the subnet the workload uses.
The destination service supports Private Google Access.
Organization policies or VPC Service Controls do not block the operation.
Production and Security Considerations
Least privilege: Private routing does not grant API access; grant only the IAM permissions the workload requires.
Egress control: Cloud NAT egress rules control which Cloud NAT external addresses are selected, including for particular destinations. Configure VPC firewall egress rules when destination restrictions are required; ingress-only firewall rules do not provide egress filtering.
Monitoring: Use Cloud Monitoring and Cloud Logging to track API errors, authorization failures, latency, quota errors, and throttling.
Service limitations: Verify Private Google Access support for each service and region; some managed services require PSC or Private Service Access.
Failure domains: Private Google Access avoids public-internet transit, but it does not eliminate service outages, quota limits, DNS failures, IAM errors, or VPC Service Controls denials.
Combined designs: A VM can use Private Google Access for supported Google services while using Cloud NAT or another approved path for general internet egress.
49
Which GCP load balancer types are available?
KnownRevise
GCP load balancers belong to two current families—Application Load Balancers for Layer 7 traffic and Network Load Balancers for Layer 4 traffic—with seven supported non-legacy configurations. The correct choice depends on protocol, geographic scope, private versus public access, routing capabilities, and source-IP preservation.
Read full answerHide full answer
Applies toGoogle Cloud Platform current stable services
Available Load Balancer Types
GCP load balancer
Traffic layer
Scope
Important characteristics
Global external Application Load Balancer
HTTP(S), gRPC
Global
Uses a global anycast IP; supports global backend services, URL maps, Cloud CDN, and Cloud Armor
Regional external Application Load Balancer
HTTP(S), gRPC
Regional
Uses a regional external IP and regional backend services; useful when regional isolation is required
Global internal Application Load Balancer
HTTP(S), gRPC
Global
Provides private Layer 7 routing across regions for internal applications
Regional internal Application Load Balancer
HTTP(S), gRPC
Regional
Provides private Layer 7 balancing within one region
External passthrough Network Load Balancer
TCP, UDP
Regional
Passes traffic directly to backends and preserves the original client source IP
Internal passthrough Network Load Balancer
TCP, UDP
Regional
Provides private Layer 4 balancing within a VPC and preserves the original client source IP
External TCP/UDP Network Load Balancer
TCP, UDP
Regional
Designed for high-throughput external traffic; does not preserve the original client source IP
For completeness, the Classic Application Load Balancer remains a documented legacy type for existing deployments. New deployments should generally use the global external Application Load Balancer instead.
How They Work
A load balancer generally consists of a forwarding rule and IP address, a backend service, health checks, and—for Application Load Balancers—a URL map and TLS configuration.
Application Load Balancers inspect HTTP or gRPC properties and route requests using host names, paths, headers, and other attributes. They support TLS termination.
Network Load Balancers operate at Layer 4 and route by IP address, port, and protocol without inspecting the application payload. They do not terminate TLS.
Supported backend types vary by product and can include managed or zonal instance groups, serverless NEGs, and individual endpoints.
Choosing the Right Type
Use the global external Application Load Balancer for public HTTP(S) applications with global users, CDN requirements, or multi-region backends.
Use the regional external Application Load Balancer when traffic and backends must remain within one region.
Use the global internal Application Load Balancer for private Layer 7 applications requiring global routing across regions.
Use the regional internal Application Load Balancer for private HTTP(S) services confined to one region.
Use an external passthrough Network Load Balancer when public TCP traffic must retain the original client IP.
Use an internal passthrough Network Load Balancer for private TCP or UDP traffic that requires source-IP preservation.
Use an external TCP/UDP Network Load Balancer for high-volume external TCP or UDP workloads where source-IP preservation is unnecessary.
AWS and Azure Equivalents
GCP type
Closest AWS equivalent
Closest Azure equivalent
Global external Application Load Balancer
Application Load Balancer combined with Global Accelerator
Front Door
Regional external Application Load Balancer
Internet-facing Application Load Balancer
Application Gateway
Global internal Application Load Balancer
No direct single-service equivalent
Front Door Premium with Private Link, depending on design
Regional internal Application Load Balancer
Internal Application Load Balancer
Internal Application Gateway
Passthrough Network Load Balancer
Network Load Balancer
Load Balancer
External TCP/UDP Network Load Balancer
Network Load Balancer
Load Balancer
These are functional comparisons rather than exact feature-for-feature equivalents.
Production Considerations
Configure health checks with the correct protocol, port, and path. A faulty check can mark every backend unhealthy and cause an outage.
Allow health-check traffic through firewall rules and restrict direct backend access where required, using proxy-only subnets where applicable.
Use managed certificates, modern SSL policies, and automated certificate renewal for HTTPS load balancers.
Monitor backend health, latency, error rates, dropped connections, utilization, and quotas in Cloud Monitoring; analyze available load-balancer logs in Cloud Logging.
Use connection draining and staged instance-group replacements during maintenance.
Plan regional capacity and inter-region data-transfer costs for globally routed backends. Global routing improves availability but does not eliminate regional capacity requirements.
Where backends need outbound internet access, plan egress through Cloud NAT or private access paths rather than assigning public IP addresses unnecessarily.
50
What is Cloud CDN in Google Cloud?
KnownRevise
Cloud CDN is Google Cloud’s global HTTP content-caching service. It stores eligible responses at Google’s edge locations so users receive cached content closer to them, reducing latency and origin load.
Read full answerHide full answer
Applies toGoogle Cloud CDN (current service)
How It Works
Cloud CDN is enabled on a backend service used by a global external Application Load Balancer. Common origins include Compute Engine instance groups, Cloud Run services, and Cloud Storage backend buckets.
For each eligible GET or HEAD request:
The load balancer routes the request to a nearby Cloud CDN point of presence.
Cloud CDN checks its cache using a cache key derived from factors such as protocol, host, path, and query parameters.
A cache hit is returned directly from the edge.
A cache miss is forwarded to the origin. The response may then be cached according to its HTTP headers and the configured cache mode.
The principal cache modes are:
USE_ORIGIN_HEADERS: Honors origin directives such as Cache-Control, Expires, and s-maxage.
CACHE_ALL_STATIC: Caches static content according to Cloud CDN’s static-content rules.
FORCE_CACHE_ALL: Caches all eligible responses even when the origin indicates otherwise; use cautiously.
Cache-Control and related response headers control how long content remains cached. Responses marked private or no-store should not be placed in a shared cache unless the design deliberately overrides that policy.
Production Behavior
Origin overload: A sudden cache miss burst can overwhelm a small origin. Origin Shield can consolidate cache fills through an intermediate location to reduce origin traffic.
Stale content: Changes remain visible until the TTL expires or an invalidation propagates. Invalidation is not instantaneous.
Cache fragmentation: Excessive query parameters or Vary headers can create many variants of the same object, reducing hit rates and increasing storage use.
Origin outage: Cached objects may continue to be served, while cache misses and expired objects fail until the origin recovers.
Eviction: Edge cache capacity is finite, so less-used objects can be evicted before their TTL. Cloud CDN is a performance layer, not the source of truth.
For mutable objects, retain the authoritative copy in Cloud Storage or another durable origin. Deploy versioned URLs or use controlled invalidation rather than deleting and immediately recreating files under the same path.
Security and Service Selection
Cloud CDN accelerates delivery but is not a replacement for a web application firewall, authentication, or origin security.
Terminate TLS at the load balancer and redirect HTTP to HTTPS.
Use Cloud Armor where supported for rate limiting, WAF rules, and DDoS defenses.
Use signed URLs or signed cookies for paid, private, or expiring content.
Restrict direct origin access where practical so users cannot bypass CDN controls.
Requirement
Google Cloud
AWS equivalent
Azure equivalent
General web and API caching
Cloud CDN
CloudFront
Azure Front Door
Compute-based origin
Compute Engine
EC2/ALB
Azure VM/Application Gateway
Serverless/API origin
Cloud Run behind external Application Load Balancer
Lambda function URL behind CloudFront
Azure Functions behind Azure Front Door
Object-storage origin
Cloud Storage backend bucket
S3 origin
Azure Blob Storage origin
Large video delivery
Media CDN
CloudFront
Azure Front Door
Use Cloud CDN for ordinary HTTP delivery. Consider Media CDN for large-scale video workflows that require specialized caching, logging, or media-oriented controls.
Monitoring and Operations
Cloud Monitoring provides load-balancer and cache metrics. Important signals include:
loadbalancer/https/backend/cdn/cache/response_count: Responses served from cache.
loadbalancer/https/backend/cdn/cache/eviction_count: Objects evicted from the edge.
loadbalancer/https/request_count: Total request volume.
A practical cache-hit ratio is cache response_count / cache lookup_count. Alert on a sustained decline, origin latency, elevated 5xx responses, and abnormal eviction rates. Enable load-balancer request logging so incident analysis can distinguish hits, misses, origin failures, and invalidation-related traffic.
Best Practices and Pitfalls
Set explicit TTLs with Cache-Control; do not rely on implicit expiration behavior.
Use immutable, versioned asset names for long-lived static files.
Keep cache keys aligned with actual representation differences.
Load-test cache misses and origin-failure scenarios, not only steady-state traffic.
Do not combine FORCE_CACHE_ALL with authenticated or personalized responses.
Monitor costs for cache fills, requests, and bandwidth separately from load-balancer charges.
Keep origin capacity sufficient for cold caches, traffic spikes, and recovery after an outage.
51
How does Google Cloud DNS manage authoritative DNS?
KnownRevise
Google Cloud DNS is Google Cloud’s managed authoritative DNS service. It serves public zones through Google’s global DNS infrastructure and private zones through the VPC DNS resolver, so users and workloads can resolve names without operating authoritative name servers or maintaining zone files.
Read full answerHide full answer
Applies toGoogle Cloud DNS (current stable managed service)
How It Works
Managed zone: A namespace such as example.com. containing resource record sets and their metadata. A public zone is visible through Google’s public DNS infrastructure; a private zone is visible only through attached VPC networks and eligible peered or shared-VPC networks.
Record sets: Entries mapping names to addresses or targets, including A, AAAA, CNAME, MX, TXT, and SRV. A zone apex cannot use a CNAME; use A, AAAA, or another supported design.
TTL: Controls how long recursive resolvers may cache an answer. A lower TTL allows changes to propagate more quickly but increases query volume because clients and resolvers refresh records more often.
Delegation: Creating a public zone does not automatically delegate it. The domain registrar must contain the Cloud DNS name-server delegation; until it does, authoritative queries do not reach the new zone.
Forwarding zones: Private forwarding zones send otherwise unmatched queries to configured internal forwarding targets, such as an on-premises DNS resolver, rather than storing those answers in Cloud DNS.
DNSSEC: Cloud DNS can sign public zones and manage the signing keys. The correct DS record must be installed at the delegation parent, normally through the registrar. Without a DS record, the delegation is treated as insecure; a stale or invalid DS record can cause validating resolvers to return SERVFAIL.
Routing policies: Public zones can use zone-level routing policies for weighted round robin, geolocation, failover, or latency-based selection. These policies are separate from DNS record types.
Cloud DNS is authoritative and is distinct from Google Public DNS, which is a recursive resolver. Cloud DNS answers from published zone data; it does not ordinarily perform recursive resolution on behalf of public-zone clients.
Production Behavior
Traffic management: Static records always return the same answer. Use Cloud DNS routing policies when selection must depend on weights, location, endpoint health, or latency. A global external Application Load Balancer is a separate mechanism: DNS resolves to its anycast IP, and the load balancer selects a backend after the connection reaches it.
Caching and rollout: DNS changes are not instantaneous because recursive resolvers can retain records for their TTL. Lower TTLs before a migration, retain old records during the transition, and restore them only after the old TTL has expired.
Monitoring: Use Cloud Monitoring for query, record-count, and other service metrics. Run external DNS probes from multiple regions to measure availability and resolution latency from the client’s perspective.
Auditability: Enable Cloud Audit Logs and retain management-operation logs for IAM, zone, record, and DNSSEC changes. Data Access logging can be enabled where read operations must also be audited. DNS query contents are not necessarily captured by configuration audit logs.
Backups and recovery: Treat zones as infrastructure as code. Store reviewed Terraform or equivalent configuration in version control, use CI validation, and retain enough historical state to recreate records after accidental deletion or configuration errors.
Security: Grant least-privilege IAM roles, scope zone-level access where appropriate, and use a dedicated service account for automation. Private DNS controls name visibility but is not an authentication boundary; applications still need IAM, mTLS, service authorization, and secrets management.
Cost: Charges can include managed zones, queries, and records. Use a zone for multiple related names, establish budgets and alerts, and avoid unnecessary zones or high query volumes.
Service Comparison
Capability
Google Cloud DNS
AWS Route 53
Azure DNS
Public authoritative DNS
Cloud DNS public managed zone
Route 53 public hosted zone
Azure DNS public zone
Private DNS
Private zone attached to VPC networks
Private hosted zone associated with VPCs
Private DNS zone linked to VNets
Health and routing
DNS routing policies
Health checks and routing policies
Traffic Manager plus Azure DNS
DNSSEC
Supported for public zones
Supported
Supported for public zones
Cloud integration
Compute Engine, GKE, Cloud Run, and multi-cloud workloads
Broad AWS and multi-cloud integration
Broad Azure and hybrid integration
Best fit
GCP-centric workloads or portable public DNS
AWS-centric estates with mature routing features
Azure-centric estates with Microsoft integrations
Common Pitfalls
Missing delegation: The zone exists, but the registrar still points to other name servers, producing NXDOMAIN or unexpected answers.
Private-zone visibility: The VPC is not attached or does not have eligible peering, or hybrid callers lack a path to the relevant VPC DNS service. Per-client IAM permission is not required for ordinary private-name resolution.
Stale caches: Records remain resolvable until cached TTLs expire; a successful API update does not erase every resolver’s cached answer.
Invalid apex CNAME: Cloud DNS does not permit a CNAME at a zone apex.
DNSSEC validation failure: A DS record exists but is stale or inconsistent with the signed zone, or a key transition was not completed correctly.
DNS treated as authentication: Restricting a name to a private zone does not replace application authorization, mTLS, or secrets management.
52
How do Cloud Interconnect and Cloud VPN provide GCP connectivity?
KnownRevise
Cloud Interconnect provides private connectivity between a Google Cloud VPC network and an on-premises network through a physical connection to Google’s network. Cloud VPN creates an encrypted IPsec tunnel between a VPC and an external VPN gateway over the public internet.
Read full answerHide full answer
Applies toGoogle Cloud current stable services (Cloud Interconnect and Cloud VPN)
How They Work
Both services extend a VPC network to on-premises or another cloud environment, but use different transports and routing models.
Cloud Interconnect
Uses a private physical connection rather than the public internet.
Dedicated Cloud Interconnect connects directly to a Google edge point of presence.
Partner Interconnect connects through a supported network provider.
A VLAN attachment connects the physical connection to a VPC.
BGP commonly exchanges routes between the VPC’s Cloud Router and the on-premises router. Static routing is also possible.
Cloud VPN
Uses IPsec with IKEv2 to encrypt traffic between the VPC and an external VPN gateway.
HA VPN creates two redundant tunnels within a region.
Dynamic routing uses BGP through Cloud Router; static routes are also supported.
HA VPN protects against a single tunnel or gateway failure, but does not automatically protect against an ISP or physical-site failure.
Service Details
Cloud Interconnect is designed for predictable throughput, low latency, and large data transfers. It avoids routing sensitive traffic over the public internet, although Cloud Interconnect does not by itself provide IPsec-style encryption. Use Cloud VPN over Interconnect or an additional encryption mechanism such as MACsec where appropriate.
A typical deployment includes:
A physical circuit or partner connection.
A VLAN attachment in a supported Google Cloud region.
A Cloud Router with a BGP peer.
Firewall rules permitting only required traffic.
Monitoring for attachment, BGP, packet, and latency conditions.
Provisioning generally requires capacity planning and lead time, especially for Dedicated Cloud Interconnect. Production installations normally use redundant attachments and diverse paths, potentially through different providers or metropolitan areas.
Cloud VPN is faster to provision and useful for moderate traffic, temporary connectivity, backup links, or locations where physical connectivity is unavailable. The external gateway must be configured with matching IKE/IPsec settings, public IP addresses, and a pre-shared key.
For new deployments, use HA VPN rather than the older single-tunnel Classic VPN design.
Comparison
Capability
Cloud Interconnect
Cloud VPN
Transport
Private physical connection
Public internet
Encryption
Not inherently provided by the interconnect
IPsec/IKEv2 encryption
Typical bandwidth
High and predictable
Moderate, dependent on internet capacity
Provisioning time
Often weeks or longer
Minutes to hours
Routing
BGP or static routes
BGP or static routes
High availability
Multiple attachments and diverse paths
Two HA VPN tunnels; additional paths for site/region failures
Common AWS equivalent
AWS Direct Connect
AWS Site-to-Site VPN
Common Azure equivalent
Azure ExpressRoute
Azure VPN Gateway
Choose Cloud Interconnect when latency, throughput, and predictable private connectivity are primary requirements. Choose Cloud VPN when rapid deployment, encrypted connectivity, or a secondary link is more important.
Production Operations
Monitor link utilization, packet loss, latency, tunnel state, and BGP session health using Cloud Monitoring and Cloud Logging.
Configure alerts for a tunnel or attachment going down, sustained route flapping, and abnormal traffic changes.
Use BGP timers and metrics appropriate to the application’s failover tolerance; overly aggressive timers can cause unnecessary route churn.
Keep routing and firewall configuration in Terraform or another infrastructure-as-code system.
Test failure scenarios deliberately by disabling a tunnel, withdrawing a route, or failing the primary ISP.
Remember that two redundant HA VPN tunnels can still share one ISP. Similarly, redundant Interconnect attachments can share the same provider or facility.
Treat VPN gateways and firewall rules as production network infrastructure: apply least-privilege IAM, change control, and regular configuration reviews.
Best Practices and Failure Modes
Use HA VPN for new designs: Two tunnels protect against tunnel and gateway failures, but use multiple regions or ISPs for broader disaster recovery.
Encrypt sensitive Interconnect traffic: A private physical path is not a substitute for application-layer or cryptographic protection.
Protect shared secrets: Store the VPN pre-shared key in Secret Manager or an equivalent secrets system, and inject it through a sensitive IaC variable rather than shell history.
Plan redundancy by failure domain: Check whether paths share power, fiber, providers, routers, or locations.
Use controlled BGP changes: Coordinate route advertisements and withdrawals to prevent accidental blackholing or traffic loops.
53
What is Google Cloud Armor and how does it protect applications?
KnownRevise
Cloud Armor is Google Cloud’s edge security service for protecting internet-facing HTTP and HTTPS applications from Layer 3/4 and Layer 7 distributed denial-of-service attacks, web application attacks, excessive requests, and bots. It combines WAF controls, rate limiting, adaptive protection, and security policy enforcement with Google’s global load-balancing infrastructure.
Read full answerHide full answer
Applies toGoogle Cloud Armor, all current versions (Standard and Enterprise tiers)
How It Works
Cloud Armor security policies attach to backend services and backend buckets behind supported external Application Load Balancers. Backend targets can include Compute Engine instance groups, serverless NEGs for Cloud Run or App Engine, and Cloud Storage backend buckets.
A typical request path is:
The client connects through a Google global anycast address.
Cloud Armor evaluates the request against the configured security policy.
A matching rule allows, denies, rate-limits, or redirects the request. Logging can be enabled for the rule.
Allowed traffic is forwarded to the application backend.
Global security policies protect the global external Application Load Balancer. Backend security policies provide a separate backend-enforcement option, particularly for regional deployments and Cloud Armor Enterprise configurations.
Cloud Armor does not replace application controls. It cannot validate business authorization, sanitize every form of input, or protect private east-west traffic that never passes through the load balancer.
Security Capabilities
Preconfigured WAF rules: Protect against common vulnerabilities using rules based on the OWASP Core Rule Set and Google-managed attack signatures.
Custom rules: Match on properties such as URL path, host, query string, HTTP method, headers, and source IP range.
Rate limiting: Restrict abusive request volume per client or another configured aggregation key, reducing application exhaustion.
Adaptive Protection: Detects abnormal Layer 7 traffic patterns and automatically applies mitigations for likely DDoS attacks; it can also recommend WAF or rate-limiting changes.
Bot management: Cloud Armor Enterprise provides advanced bot detection and management for scenarios such as credential-stuffing attacks and resource scraping, and can integrate with reCAPTCHA Enterprise to challenge suspicious clients.
DDoS protection: Combines Google’s network infrastructure with Layer 7 controls designed to prevent application overload.
Rules are evaluated in policy order. Administrators should place narrowly targeted deny and rate-limit rules before broad allows while keeping logging and default behavior understandable.
Production Operations
Cloud Armor observability comes from Cloud Logging and Cloud Monitoring. When request logging is enabled, security-policy logs help identify denied traffic, matched rules, client IP addresses, user agents, and other request attributes. Useful operational signals include:
Sudden increases in denied requests
Unexpected changes in allowed request volume
Backend latency or error-rate increases
Repeated rate-limit matches
WAF matches associated with a known vulnerability
Legitimate users being blocked after a rule or application change
Policy changes should follow the same controls as other production changes: test in a non-production policy, review the rule scope, monitor immediately after rollout, and maintain a rollback path. Broad exclusions can silently weaken WAF protection and should be removed once the underlying issue is fixed.
Failure Modes and Security Boundaries
Bypassing the load balancer: If attackers can reach a backend directly, Cloud Armor cannot inspect that traffic. Use private backends, restrictive VPC firewall rules, and no unnecessary public IP addresses.
Overly broad allow rules: An early allow rule can bypass later WAF or rate-limit controls. Understand rule precedence before deployment.
Poor rate-limit keys: Limiting by a shared NAT address may block many legitimate users, while easily spoofed or overly specific keys may provide little protection.
WAF false positives: Business applications using unusual parameters or workflows can trigger legitimate traffic. Add narrow, documented exclusions rather than disabling protection.
Application-layer abuse: Cloud Armor reduces harmful traffic, but it does not fix insecure authorization, exposed secrets, or application vulnerabilities.
Availability dependency: Internet protection depends on traffic using the protected load-balancing path. Test failover, health checks, and incident runbooks rather than assuming the edge eliminates all availability risks.
AWS, Azure, and GCP Comparison
Capability
Google Cloud
AWS
Azure
Managed WAF and DDoS protection
Cloud Armor
AWS WAF + AWS Shield
Azure Front Door WAF + DDoS Protection
Primary protected entry point
External Application Load Balancer
CloudFront, ALB, or regional services
Front Door, Application Gateway, and related services
Bot management
Cloud Armor Enterprise
AWS WAF Bot Control
Azure Bot Protection
Private east-west traffic
Cloud NGFW and VPC firewall controls
Network Firewall, Security Groups, and VPC controls
Azure Firewall and NSGs
Quotas and pricing
Tiered feature and policy limits
Per-rule, request, and Shield pricing
Plan- and rule-based pricing
Choose Cloud Armor when a GCP application is exposed through an external load balancer and needs integrated WAF, DDoS mitigation, rate limiting, and edge policy enforcement. Use Cloud NGFW or VPC firewall controls for private network segmentation, and implement authorization and validation inside the application itself.
54
What is Google Cloud Identity-Aware Proxy (IAP)?
KnownRevise
Identity-Aware Proxy (IAP) is a Google Cloud service that provides identity-based access control for HTTP(S) applications and IAM-authorized TCP tunneling for SSH access to Compute Engine instances. For web applications, IAP performs authentication and authorization before the request reaches the backend, while the application can validate IAP’s signed identity assertion.
Read full answerHide full answer
Applies toGoogle Cloud IAP (all currently supported versions)
How It Works
IAP commonly sits between a browser or client and a backend application, often behind an external Application Load Balancer:
A user requests an IAP-protected HTTPS application.
If the user has not authenticated, IAP redirects the browser to the configured identity flow, which can use Google sign-in or a federated external identity provider.
IAP evaluates the user’s Cloud IAM permissions for the protected resource.
If access is allowed, IAP forwards the request and adds identity headers, including the signed X-Goog-IAP-JWT-Assertion header.
If access is denied, IAP blocks the request before it reaches the backend.
For SSH, IAP TCP forwarding creates an encrypted tunnel from the client to a Compute Engine instance. IAM controls who may establish the tunnel through the roles/iap.tunnelResourceAccessor role, but IAP does not replace SSH authentication: the client still performs normal SSH host and user authentication after the tunnel is established. This allows SSH to work without exposing port 22 directly to the internet.
IAM and Identity
IAP separates identity verification from resource authorization:
Authentication: The identity provider verifies the user, using Google Workspace, Cloud Identity, or a configured federated identity provider.
Web authorization: roles/iap.httpsResourceAccessor permits a principal to access IAP-protected web applications.
Tunnel authorization: roles/iap.tunnelResourceAccessor permits a principal to establish an IAP TCP tunnel.
Service agent: roles/iap.serviceAgent gives the IAP service agent the permissions it needs to operate on protected resources.
Applications exposed to untrusted networks should validate the signed IAP JWT assertion, including its signature, expected audience, expiry, and relevant identity claims. The simpler authenticated-user email header must not be trusted unless the backend can guarantee that requests cannot bypass IAP.
Supported Scenarios
Capability
Google Cloud
AWS equivalent
Azure equivalent
Application access control
IAP with Cloud IAM
ALB authentication with Cognito or another OIDC provider
App Service Authentication with Microsoft Entra ID
Browser-based sign-in
Google Workspace, Cloud Identity, or federated identity provider
Cognito or an external OIDC provider
Microsoft Entra ID or an external OAuth/OIDC provider
SSH access
IAP TCP forwarding plus normal SSH authentication
SSM Session Manager
Azure Bastion
Authorization source
Cloud IAM policies
IAM for AWS resources; application authorizers or application logic for protected apps
Azure RBAC; application authentication and authorization for application data
These products are not exact one-to-one equivalents. In particular, Cognito and Entra ID primarily provide identity, while an application or authorizer usually decides what an authenticated user can access.
Security and Production Considerations
Prevent bypass: Do not give users a direct route to the backend. Use private backends where possible. On Compute Engine, permit only the applicable external Application Load Balancer and health-check paths; for IAP SSH forwarding, allow TCP port 22 from IAP’s 35.235.240.0/20 range.
Do not trust arbitrary headers: A client can forge X-Goog-Authenticated-User-Email. Block direct access or validate X-Goog-IAP-JWT-Assertion.
IAP is not a WAF: Use Cloud Armor for edge web-application and DDoS controls, plus application-level validation, rate limiting, and abuse prevention.
Machine clients need a different pattern: Use IAM-authenticated service accounts, Workload Identity Federation, or service-specific authentication rather than treating IAP as a machine-to-machine gateway.
Availability dependency: IAP depends on Google Cloud and the configured identity provider. Monitor load-balancer health, backend latency, relevant logs, and identity-provider failures.
Audit access: Enable appropriate Cloud Audit Logs and review changes to IAP settings and IAM bindings, especially changes to roles/iap.httpsResourceAccessor and roles/iap.tunnelResourceAccessor.
Best Practices and Failure Modes
Grant roles/iap.httpsResourceAccessor at the narrowest practical resource scope rather than project-wide.
Use groups or workforce identities for onboarding and offboarding instead of individual user bindings.
Test successful authentication, denied IAM access, expired sessions, and direct-backend bypass attempts.
Validate the signed IAP JWT in sensitive applications rather than relying only on email headers.
Monitor load-balancer logs and metrics, audit IAM changes, and correlate authentication failures with backend health.
For IAP-forwarded SSH, continue managing sshd, key access, patching, and OS-level logging; IAP protects the network path but does not secure the guest operating system.
55
What is Google Cloud Secret Manager?
KnownRevise
Google Cloud Secret Manager is a managed service for securely storing, controlling, and retrieving sensitive values such as passwords, API keys, and certificates. It provides encrypted storage, versioning, IAM-based access control, replication, and audit logging without putting secrets directly in application code or container images.
Read full answerHide full answer
Applies toGoogle Cloud Secret Manager (all currently supported versions)
How It Works
Secrets and versions: A secret is a logical resource, while each stored value creates an immutable version. A secret can have up to 100 versions.
Replication policy: Secret resources use a global API, but their payloads are replicated according to the secret’s policy:
automatic replicates data across Google Cloud regions for higher availability.
user-managed selects specific replication locations.
Version states: Versions can be enabled, disabled, or destroyed. A disabled version cannot be accessed normally but can be re-enabled; a destroyed version cannot be restored. Audit records remain available.
Payload limit: A single secret version supports up to 64 KiB of secret data.
Encryption: Data is encrypted at rest by default. Customer-managed encryption keys through Cloud KMS can be configured when additional key-control requirements exist.
Size and format: Secret Manager stores arbitrary values, commonly text, JSON, certificates, or short configuration blobs.
Security and IAM
Secret Manager uses Google Cloud IAM at the organization, folder, project, or individual secret level. Access to secret data and administrative operations are separate permissions.
The commonly used role is:
roles/secretmanager.secretAccessor: Allows the principal to access secret payload versions.
A production application should normally receive only roles/secretmanager.secretAccessor on the specific secrets it needs. Avoid granting broad administrative roles to runtime identities.
Service accounts are recommended for Compute Engine, GKE, Cloud Run, and other hosted workloads. Avoid downloading the secret manually and embedding it in deployment configuration. In GKE, the Secrets Store CSI Driver can mount Secret Manager secrets into workloads.
Secret Manager supports TLS in transit, and retrieving metadata does not reveal the secret payload. Secret values remain sensitive if copied into logs, shell history, CI output, Terraform state, or crash reports.
Production Behavior
Version updates: Adding a version does not update consumers that already read or cached its value. An explicitly referenced version remains fixed, while a consumer that reads the latest alias on each request receives the newest enabled version at the next successful read.
Controlled rotation: Add and validate a new version before consumers adopt it. If production consumers use latest, adding the version changes that alias immediately, so use pinned versions or a deployment-controlled configuration when a staged rollout is required. For irreversible rollback, add the previous value again as a new version rather than assuming a destroyed version can be restored.
Availability: Treat Secret Manager access as a startup dependency. Applications should handle transient failures with bounded retries, timeouts, and clear readiness checks.
Monitoring: Enable and retain Secret Manager Data Access audit logs where access visibility is required. Monitor version creation, IAM changes, denied requests, and unusual access patterns through Cloud Logging and Security Command Center.
Incident response: If a value is compromised, add a replacement version, update consumers, and then disable or destroy the exposed version. Removing a secret alone does not update already-running applications.
Service Comparisons
Need
Google Cloud
AWS
Azure
Application secrets
Secret Manager
Secrets Manager
Key Vault
Encryption-key management
Cloud KMS
KMS
Key Vault or managed HSM
General secret types
Primarily secret values
Secrets, binary values, and related metadata
Secrets, keys, and certificates
Typical GCP choice
Secret Manager for credentials; Cloud KMS for keys
—
—
Secret Manager is appropriate when an application needs a managed secret payload with IAM and versioning. Cloud KMS is different: it manages cryptographic keys used for encryption, signing, and envelope encryption rather than application passwords or API tokens.
Best Practices and Failure Modes
Use least-privilege IAM and separate deployment identities from runtime identities.
Use secret references, not literal values, in Cloud Run configuration, Kubernetes manifests, and CI variables.
Never print payloads during routine diagnostics; verify access through metadata and audit logs instead.
Require rotation procedures that add, validate, activate, and retire versions without causing coordinated downtime.
Handle 403 errors by checking IAM bindings and workload identity, and 404 errors by checking the project, secret name, and replication location.
Handle disabled or destroyed versions explicitly; restarting the application does not make them accessible again.
Keep audit logs in a protected log sink, because application logs and audit records can be deleted or altered independently.
56
What are Cloud KMS and customer-managed encryption keys in GCP?
KnownRevise
Cloud KMS is Google Cloud’s centralized service for creating, protecting, rotating, and using cryptographic keys without exposing their raw key material. A customer-managed encryption key (CMEK) is an encryption key whose use and lifecycle are controlled by the customer rather than through Google-managed encryption keys.
Read full answerHide full answer
Applies toGoogle Cloud current stable services (Cloud KMS API v1)
Core Concepts
Cloud KMS organizes keys into a hierarchy:
Key ring: A logical container for related cryptographic keys and an administrative boundary. A key ring has a fixed location.
Crypto key: A named key resource containing metadata, an IAM policy, and one or more cryptographic key versions.
Key version: An immutable set of cryptographic key material. Rotation creates a new version instead of changing an existing one.
Location: Regional, multi-region, and global locations are supported. A key cannot be moved after creation, and its location must be compatible with the resource using it.
Purpose: Restricts allowed operations, including symmetric encryption, asymmetric encryption, digital signatures, and HMAC operations.
Protection level: Determines where cryptographic operations are performed, including SOFTWARE, HSM, EXTERNAL, and EXTERNAL_VPC protection where supported.
Cloud KMS is comparable to AWS KMS and Azure Key Vault. The closest AWS concept is a customer-managed key in AWS KMS; Azure customer-managed keys are commonly backed by Azure Key Vault.
How CMEK Encryption Works
Managed Google Cloud services typically use envelope encryption:
The service generates a random data encryption key (DEK).
Cloud KMS encrypts the DEK with the CMEK.
The service stores the encrypted DEK with the encrypted resource or its metadata.
When the resource is read, the service requests decryption of the DEK through Cloud KMS and then decrypts the data.
Cloud KMS authorizes operations through IAM and returns cryptographic results rather than exporting the underlying key material. For standard symmetric keys stored in Cloud KMS, key material is not exportable.
CMEK and Other Key Models
Model
Management and control
Operational trade-off
Google-managed encryption keys
Google manages key protection and rotation; customers have little direct lifecycle control
Simplest, but customers cannot independently rotate or disable the key
CMEKs in Cloud KMS
Customers control key IAM, rotation, disabling, destruction, and location
Greater control and separation of duties, with additional IAM, quota, and availability dependencies
CMEKs backed by Cloud HSM or Cloud External Key Manager
Customers retain control in dedicated HSMs or supported external key systems
Stronger custody or sovereignty requirements, but service and key-type support can be narrower
Customer-supplied encryption keys (CSEK), where supported
Customers generate, protect, and supply key material to the consuming service
Avoids reliance on Cloud KMS for cryptographic operations but creates key-delivery, storage, recovery, and rotation burdens
A CMEK changes control over the key lifecycle, but it does not remove dependence on Google Cloud. IAM errors, key-location restrictions, KMS quotas, and service outages can still make protected data unavailable.
Security and Operational Behavior
IAM is the authorization boundary: The predefined role roles/cloudkms.cryptoKeyEncrypterDecrypter grants symmetric encryption and decryption operations. Production grants should be limited to the identities and service agents required by each service.
Managed-service identities matter: For service-encrypted CMEK integrations, grant the exact service agent or service identity specified in that service’s documentation. Depending on the service, the calling principal may also require KMS permissions; a runtime identity’s direct grant does not automatically grant a managed service permission to use the key.
Rotation is forward-looking: A new primary version handles new encryption. Existing ciphertext remains associated with the key version that protected its DEK and is not automatically re-encrypted.
Disabling can affect existing data: Disabling a key version prevents cryptographic operations with that version, including decryption of data that still depends on it. Consumers may receive authorization or availability errors.
Destruction is hazardous: A disabled key version can be scheduled for destruction. Destruction can be canceled while the version is scheduled (30 days by default, configurable from 24 hours to 120 days), but a destroyed version cannot be restored. Do not assume Cloud KMS will prevent destruction because another Google Cloud resource still references the key.
Audit usage carefully: Cloud Audit Logs records KMS administration. Data Access logging must be enabled separately if decrypt-operation visibility is required. Monitor key administration, IAM changes, disabled keys, authorization failures, latency, and quota consumption without logging plaintext or key material.
Production Best Practices
Use separate keys for production, non-production, and distinct security or compliance boundaries.
Manage key rings, key versions, IAM bindings, and locations through reviewed infrastructure as code.
Require approval for key or version destruction and use lifecycle safeguards such as Terraform prevent_destroy.
Re-encrypt stored data deliberately before retiring older key versions.
Validate that service identities, key locations, IAM conditions, and KMS quotas match each consumer’s requirements.
Maintain tested recovery and incident runbooks for disabled keys, mistaken IAM changes, pending destruction, quota exhaustion, and regional failures.
57
What is VPC Service Controls in Google Cloud?
KnownRevise
VPC Service Controls is a Google Cloud security perimeter that limits how supported Google Cloud resources can be accessed and where their data can be sent. It works alongside IAM, protecting organizations from unauthorized access and data exfiltration across project or organizational boundaries.
Read full answerHide full answer
Applies toGoogle Cloud VPC Service Controls (current)
Core Concepts
A service perimeter defines a protected boundary for Google Cloud resources.
Access policy: The organizational policy container that owns service perimeters, access levels, and perimeter bridges.
Service perimeter: A boundary associated with an access policy and the projects it protects.
Protected resources: Resources inside the perimeter, such as a project containing Cloud Storage buckets or BigQuery datasets.
Restricted services: Google Cloud services protected by the perimeter. Support varies by service, so verify that each resource type is covered.
Access level: Grants specified users or groups permission to cross the boundary under configured conditions. It is an explicit exception, not a general public-access setting.
Perimeter bridge: Allows resources in one perimeter to communicate with resources in another perimeter.
VPC Service Controls is not a replacement for IAM. IAM still determines what an identity is authorized to do, while VPC Service Controls determines whether the request is allowed to cross the security boundary at all.
How Enforcement Works
For a request involving a protected service, Google Cloud evaluates the request’s identity and available access context against the perimeter’s policies.
Ingress evaluation: Checks whether the requester is inside the perimeter or has an applicable access level or other explicit exception.
Identity authorization: IAM still evaluates the caller’s roles and permissions.
Egress evaluation: Checks where a resource inside the perimeter is permitted to send data.
Logging: Access Context Manager audit logs and Cloud Audit Logs can show denied or allowed perimeter activity.
Ingress and egress rules should normally be restrictive. For example, a perimeter can allow workloads in a production project to read a specific BigQuery dataset while preventing arbitrary external access. Egress rules are especially important because unrestricted egress can allow sensitive data to leave the boundary.
VPC Service Controls applies primarily to supported Google Cloud service APIs and resources. It is not a general-purpose firewall, does not inspect arbitrary network traffic, and does not prevent an authorized identity from reading data that it is already permitted to access.
GCP and Cloud Comparisons
Capability
Google Cloud
AWS
Azure
Perimeter or policy boundary
VPC Service Controls
Organizations SCPs and related controls
Azure Policy and Management Groups
Network path restriction
Service perimeter and related networking controls
VPC endpoints, PrivateLink, and security groups
Private Link, private endpoints, NSGs, and Azure Firewall
Main purpose
Prevent unauthorized Google Cloud access and data exfiltration
Control API actions and network access
Control resource behavior and network access
Closest equivalent
No exact one-to-one equivalent
SCPs plus PrivateLink or VPC controls
Azure Policy plus Private Link or Firewall
Choose VPC Service Controls when the primary requirement is a Google Cloud data-access boundary, such as isolating production analytics data from development identities or external contractors. Use IAM, VPC firewall rules, private connectivity, and organization policies alongside it.
Production Best Practices and Failure Modes
Start in a non-production environment: Test every client, integration, CI/CD job, and support workflow before applying the perimeter broadly.
Use narrow access levels: Grant access to specific identities or groups and use conditions where possible. Treat access levels as production change objects.
Avoid wildcard egress rules: Allow only required destinations and services, then monitor unexpected access-denied events.
Monitor continuously: Review Access Context Manager audit logs, Cloud Audit Logs, Security Command Center findings, and application errors for perimeter violations.
Plan for failures: A missing access level, incorrect resource membership, unsupported service, or overly restrictive egress rule can interrupt production workloads.
Protect the boundary itself: Limit who can modify access policies, perimeters, access levels, and bridges. VPC Service Controls does not replace encryption, DLP, least-privilege IAM, or data-loss prevention controls.
Plan recovery and continuity: Keep tested break-glass access for authorized responders, but audit its use. VPC Service Controls provides no backup or data-recovery capability.
58
What is Google Cloud Security Command Center?
KnownRevise
Google Cloud Security Command Center (SCC) is Google Cloud’s centralized security posture and threat-detection service. It brings asset inventory, vulnerability findings, misconfiguration detection, threat detection, sensitive-data discovery, and compliance capabilities into one view.
Read full answerHide full answer
Applies toGoogle Cloud Security Command Center (Standard, Premium, and Enterprise editions; current service)
Core Capabilities
SCC helps security and operations teams identify and prioritize risk across Google Cloud resources.
Asset visibility: Discovers Google Cloud assets and organizes them by organization, folder, project, and resource type.
Security posture: Identifies misconfigurations involving IAM, public access, encryption, networking, and other supported controls.
Vulnerability management: Reports vulnerable software, container images, and deployed workloads for supported resources.
Threat detection: Uses security telemetry and logs to identify suspicious behavior, including anomalous service-account activity, malware, and cryptocurrency mining.
Sensitive-data discovery: Identifies sensitive information in supported data stores. Access Transparency Analyzer separately provides access-risk insights where available.
Compliance management: Evaluates resources against supported benchmarks and frameworks and provides posture findings and compliance reports.
Attack-path intelligence: Shows how supported weaknesses and excessive permissions might form a reachable attack path.
Availability depends on the configured edition, enabled modules, resource coverage, and regional support.
How It Works
SCC combines Google Cloud resource metadata with security telemetry generated by Google Cloud services.
Discovery uses Google Cloud Asset Inventory and other resource information to identify assets and their configurations.
Detection modules inspect configurations, vulnerability information, identity activity, network behavior, and service logs.
Findings identify the affected resource, risk category, severity, evidence, and remediation guidance.
Notification and export sends findings through Pub/Sub and supported integrations such as SIEM or ticketing workflows.
Attack-path analysis, where supported, relates weaknesses across resources to reveal exploitable chains.
A finding is a risk signal, not proof of exploitation. For example, a public-bucket finding still requires investigation to determine whether the bucket contains sensitive data and was intentionally exposed.
Production Operations
A production deployment should integrate SCC with existing SRE, security, and incident-response workflows.
Alert on new high-severity findings affecting production, internet-facing, identity, or regulated resources.
Group findings by project, resource, category, and business owner.
Create tickets containing the affected resource, evidence, remediation guidance, owner, and escalation deadline.
Measure finding age, recurrence, and closure quality rather than tracking only alert volume.
Export relevant findings through Pub/Sub and enrich them with asset ownership.
Apply deduplication, severity normalization, suppression, and documented risk-acceptance rules.
Active organization-level findings can be reviewed with:
gcloud scc findings list --organization=ORGANIZATION_ID --state=ACTIVE
SCC does not replace Cloud Audit Logs, Cloud Logging, or a full SIEM. It adds security-focused findings and context to those sources.
IAM and Organizational Scope
SCC is normally activated at the organization level to provide centralized coverage across folders and projects.
Grant least-privilege access for finding triage, finding management, and security administration.
Restrict broad viewer access because findings can expose infrastructure and vulnerability details.
Separate users who investigate findings from users who administer organization-wide security settings.
Align access with the organization’s folder and project ownership model.
Monitor SCC configuration changes through Cloud Audit Logs.
Keep authoritative remediation in the underlying resource rather than relying on finding state as proof that the risk is fixed.
Comparison With AWS and Azure
Platform
Central security platform
Native threat detection
Vulnerability management
Sensitive-data discovery
Google Cloud
Security Command Center
SCC threat modules
SCC vulnerability findings
Edition- and resource-dependent
AWS
AWS Security Hub
AWS GuardDuty
Amazon Inspector
AWS Macie
Azure
Microsoft Defender for Cloud
Defender for Cloud plans
Defender integrations
Defender or Purview, depending on plan and service
Choose SCC when a team wants a unified Google Cloud view spanning IAM posture, vulnerabilities, threats, compliance, and security operations. The AWS and Azure comparisons also combine several related products rather than relying on one service for every capability.
Best Practices and Common Pitfalls
Validate coverage: Compare SCC’s asset inventory with expected projects, regions, and services.
Prioritize actionable findings: Focus first on internet-facing, production, identity, and data-bearing resources.
Tune carefully: Validate intentional configurations and document accepted exceptions before suppressing alerts.
Protect finding data: Restrict access because findings reveal resource names, vulnerabilities, and relationships.
Check boundaries: Confirm detector availability, edition requirements, resource support, and regional behavior before relying on a finding type operationally.
59
How do Google Cloud Logging and log sinks work?
KnownRevise
Google Cloud Logging is Google Cloud’s managed service for ingesting, storing, querying, and monitoring logs from GCP resources, applications, and supported agents. A log sink is a filtered routing rule that copies matching log entries to another destination while leaving the original entries in Cloud Logging.
Ingestion: Logs arrive from services such as Compute Engine, GKE, Cloud Run, and Cloud Storage, or from the Ops Agent, Cloud Logging API, and application stdout/stderr.
Organization: Each entry contains a logName, resource.type, severity, timestamps, labels, and text or structured payload fields.
Querying: Logs Explorer and the Logging API use filters based on fields such as severity, resource.type, and logName.
Retention: The automatically created _Default bucket normally retains logs for 30 days, while required logs in _Required are retained for 400 days. Retention can be changed where permitted.
Export: Sinks create independent copies; exporting does not remove or shorten the source log’s retention.
How Sink Routing Works
A sink defines a destination, optional filter, and dedicated writer identity.
Incoming entries are evaluated against each sink’s filter.
Every matching sink receives a copy of the entry.
The writer identity authenticates to the destination and writes the exported records.
Sinks can be owned by an organization, folder, or project. A project sink includes that project’s child resources by default but does not capture logs from other projects. Routing is asynchronous and should not be treated as an exactly-once or ordered messaging system. Destination failures can delay exports, and organizations should not rely on sink delivery for transactional correctness.
Destination Choices
Destination
Typical use
Sink writer access
AWS/Azure equivalent
Cloud Storage
Low-cost, long-term archives
roles/storage.objectCreator
Amazon S3 / Azure Blob Storage
Pub/Sub
Event streaming and custom processing
roles/pubsub.publisher
Amazon Kinesis or Firehose / Azure Event Hubs
BigQuery
SQL-based log analytics
roles/bigquery.dataOwner for sink-managed dataset/table creation; scoped roles/bigquery.dataEditor for an existing target
Athena / Azure Log Analytics
Cloud Logging bucket
Separation of retention or access boundaries
roles/logging.bucketWriter on the destination bucket
CloudWatch log groups / Log Analytics workspaces
Cloud Storage is usually best for retention, Pub/Sub for real-time processing, and BigQuery for structured analysis. AWS CloudWatch Logs and Azure Monitor Logs provide similar central logging, but their export mechanisms are subscription filters and diagnostic settings respectively.
Code Example
The following creates a bucket, exports Admin Activity audit logs, and grants the sink permission to write objects:
The gcloud logging sinks create command creates a dedicated service account automatically when one is not supplied. For Pub/Sub, grant that identity roles/pubsub.publisher on the topic. For BigQuery, use roles/bigquery.dataOwner when the sink must create the dataset and table; for a pre-existing target, grant the narrowest applicable dataset or table permissions, commonly roles/bigquery.dataEditor.
Production, Security, and Failure Modes
Monitor delivery: Alert on sink errors and monitor destination-specific signals such as Pub/Sub backlog, failed object writes, and BigQuery ingestion failures.
Control permissions: roles/logging.configWriter can redirect project logs, so grant it narrowly. Review sink changes as security-sensitive configuration.
Protect archives: For Cloud Storage, consider retention policies, bucket versioning, object lock, customer-managed encryption keys, and lifecycle rules.
Avoid sensitive data: Logs commonly contain tokens, request bodies, and user data. Redact secrets before ingestion rather than relying on downstream filtering.
Plan capacity and cost: Storage objects, BigQuery streaming ingestion, network transfer, and longer retention all increase cost. Use targeted filters and lifecycle controls.
Design for failure: A sink is a secondary export path, not a guaranteed backup. Keep primary logs available through Cloud Logging and validate archived records periodically.
60
How does Google Cloud Monitoring work, and how are alerts created?
KnownRevise
Cloud Monitoring is Google Cloud’s time-series observability service. It ingests metrics from Google Cloud services, agents, and custom applications, stores them for analysis, powers dashboards, and evaluates alert policies that notify operators when conditions are breached.
Read full answerHide full answer
Applies toGoogle Cloud Monitoring (current) and Google Cloud CLI (current stable)
How Monitoring Works
Google Cloud services automatically publish metrics such as CPU utilization, request latency, database saturation, and error counts. Agents and applications can publish custom business and application metrics.
Metric: A measured value, such as compute.googleapis.com/instance/cpu/utilization.
Time series: The values of one metric for one monitored resource and label combination.
Monitored resource: The object producing the metric, such as a Compute Engine instance or Cloud Run revision.
Dashboard: A visualization combining charts, metrics, and layout information.
Alerting policy: A rule containing one or more conditions and zero or more notification channels. Conditions can evaluate thresholds, missing data, or matched logs.
Monitoring policies are evaluated periodically against recently received data. Delayed or missing telemetry can therefore delay detection or prevent a normal threshold condition from firing. Cloud Logging log-based metrics also appear as Monitoring time series, but Cloud Monitoring is primarily a metrics service rather than a full log-management system.
How Alert Evaluation Works
A threshold condition selects time series with a filter, aggregates their data, and compares the result with a threshold. For example:
metric.type = "compute.googleapis.com/instance/cpu/utilization"
AND resource.type = "gce_instance"
A condition can specify:
Comparison: Such as COMPARISON_GT for greater than.
Threshold: The value that triggers the condition.
Duration: How long the condition must remain true.
Aggregation: An alignment period and function such as ALIGN_MEAN.
Absence condition: Fires when matching time series stop reporting data.
Each matching time series is evaluated independently unless a cross-series reduction is configured. An absence condition can detect missing data for a series previously observed, but it cannot detect a resource that never produced any time series.
Multiple conditions can be combined with AND or OR. This supports multi-window, multi-burn-rate alerting for SLOs, which is less noisy than a single low threshold.
Code Example
First, identify a verified notification channel:
gcloud monitoring channels list --project=example-project
Then create an alerting policy that alerts when the 60-second mean CPU utilization exceeds 80% for five minutes:
gcloud monitoring alerts create \
--project=example-project \
--display-name='GCE CPU above 80 percent' \
--criteria='metric.type="compute.googleapis.com/instance/cpu/utilization" AND resource.type="gce_instance" AND resource.labels.zone="us-central1-a"' \
--aggregation='alignmentPeriod=60s,perSeriesAligner=ALIGN_MEAN' \
--conditions='condition=conditionThreshold,comparison=COMPARISON_GT,thresholdValue=0.8,duration=300s' \
--notification-channels='projects/example-project/notificationChannels/1234567890' \
--enabled
CPU utilization is expressed as a fraction from 0 to 1, so 0.8 means 80%. The filter covers every matching instance in the specified zone; restrict it with resource.labels.instance_id to target one instance.
Use service-level signals: Prefer user-visible symptoms such as latency and error rate over isolated resource metrics.
Add a duration: A five-minute threshold is generally less noisy than an instantaneous spike.
Avoid alert fatigue: Give every alert an owner, severity, runbook, and clear response action.
Monitor the alert path: Track notification-channel failures, disabled policies, missing metrics, and pending channel verification.
Use aggregation carefully: Cross-series reduction can combine related instances, but it can also conceal failures affecting only a subset.
Prefer burn-rate alerts for SLOs: They balance fast detection against sustained error-budget consumption.
Control cardinality: Excess labels can consume the active time-series quota and increase costs for billable custom metrics.
Use complementary checks: Metric alerts detect abnormal measurements, absence alerts detect missing expected telemetry, and uptime checks test configured synthetic endpoints.
Security and Failure Modes
Grant roles/monitoring.viewer for read access and roles/monitoring.editor only where alert policies must be changed; use custom roles for tighter privilege control.
Scope permissions to the narrowest practical folder, project, or monitored resource.
Verify email and SMS recipients, and protect webhook channels with appropriate authentication.
Store webhook credentials in an approved secret-management system rather than scripts or source control.
Remember that healthy monitoring infrastructure does not guarantee healthy applications: agents, permissions, quotas, and metric pipelines can all fail.
Combine alerts with dashboards, logs, traces, and incident-management processes. A notification alone does not establish severity, diagnosis, or ownership.
61
How do Cloud Trace and Cloud Profiler work in Google Cloud?
KnownRevise
Cloud Trace provides distributed tracing for request lifecycles across services, while Cloud Profiler samples execution stacks to identify CPU, memory, allocation, and contention hotspots. They are complementary: tracing identifies a slow request path, and profiling shows where a workload consumes resources around the same time, although the two datasets are not directly linked at the individual-span level.
Read full answerHide full answer
Applies toGoogle Cloud Trace API v2 and current supported Cloud Profiler agents; Cloud Profiler’s managed agent has no public semantic version
Core Difference
Cloud Trace: Records parent and child operations as spans, including timestamps, durations, attributes, status, and identifiers such as trace_id and span_id.
Cloud Profiler: Periodically captures stack samples and aggregates them into CPU, heap, allocation, contention, or thread profiles.
Combined use: A trace may show that POST /checkout consistently takes 800 ms. A CPU profile collected from the same service and version may then reveal CPU-heavy serialization or encryption. It will not expose time spent waiting inside a remote database unless that wait is separately visible through contention or other telemetry.
Cloud Trace is primarily a request-oriented latency-analysis tool. Cloud Profiler is a continuous, workload-oriented performance-analysis tool whose profiles represent aggregated activity rather than one request.
How They Work
Cloud Trace combines automatic and instrumented telemetry:
Some supported Google Cloud services provide trace data automatically.
Google Cloud client libraries and supported OpenTelemetry instrumentation create application spans.
W3C Trace Context, commonly carried in the traceparent header, propagates trace context between services.
Exported spans are grouped into traces by trace ID. With OpenTelemetry, an OpenTelemetry Collector can receive OTLP spans and export selected traces to Google Cloud.
Span attributes can identify the service, operation, RPC method, HTTP route, status code, location, and associated resource.
Cloud Profiler uses an agent running with the workload:
The agent periodically samples stacks and uploads aggregated profiles.
Profiles are associated with the service, service version, monitored resource, and profile type.
CPU profiles identify sampled CPU time and its call paths.
Heap profiles show allocations made since the previous heap-profile collection, while allocation profiles show allocation activity over their configured interval.
Contention profiles identify time spent waiting for locks and other synchronization primitives.
Thread profiles provide a view of activity across an application’s threads.
Neither product replaces structured logging or metrics. Traces provide request-level causality, profiles provide workload-level code evidence, and Cloud Monitoring supplies fleet-wide signals and SLO measurements.
Service Comparisons
Google Cloud service
AWS equivalent
Azure equivalent
Important distinction
Cloud Trace
AWS X-Ray
Azure Application Insights
Distributed request tracing and dependency analysis
Cloud Profiler
AWS CodeGuru Profiler
Azure Monitor Profiler
Cloud Profiler supports CPU, heap, allocation, contention, and thread profiling; the cloud products differ in instrumentation, analysis interfaces, sampling, retention, and pricing
Migration should preserve instrumentation concepts rather than assume that configuration, permissions, or sampling behavior is identical across providers.
Production Practices
Sampling: Use parent-based trace sampling to control volume. Head sampling can discard rare errors before they are known, so use collector-side tail sampling to retain errors, high-latency requests, and other interesting traces when collector resources permit.
Security: Treat trace attributes and profile labels as potentially sensitive. Never record tokens, credentials, personal data, raw request bodies, or unrestricted query strings. Use application-level allowlists and redaction.
Overhead: Benchmark trace-export and profiler-agent overhead under peak load. Monitor CPU usage, collector memory pressure, dropped spans, upload failures, and ingestion volume.
Observability: Correlate trace IDs with structured logs. Alert on failed exports, unhealthy profiler agents, abnormal trace volume, and service-level objective regressions.
Incident response: Capture profiles while the symptom is active, then compare profiles across failing and healthy versions or regions. Keep trace sampling and deployment changes controlled so incident evidence is not obscured.
Cost control: Remove unused attributes, tune trace sampling, and monitor billing separately for trace ingestion or retention and Cloud Profiler processing.
Retention: Treat traces and profiles as diagnostic data with managed quotas and retention periods, not as permanent audit or compliance records.
62
How does Google Cloud Build automate CI/CD workflows?
KnownRevise
Google Cloud Build is Google Cloud’s managed CI/CD service. It fetches source code, executes build steps in isolated environments, publishes artifacts such as container images, and records build status and logs for deployment workflows.
A build begins through a repository trigger or a manual command such as gcloud builds submit:
Cloud Build retrieves the requested source revision.
It provisions the required build environment.
Each step runs in an isolated container. Steps share the build workspace, allowing later steps to consume files produced by earlier ones.
The build pushes images or other artifacts to services such as Artifact Registry or Cloud Storage.
Cloud Build publishes the final status and logs to Cloud Logging.
A typical pipeline compiles an application, runs tests, builds a container image, scans it, and publishes it. A nonzero exit code fails the build, and subsequent dependent steps do not run.
Capabilities and Design Choices
cloudbuild.yaml: Defines steps, images, substitutions, artifacts, and build options.
Build steps: Run shell commands, Docker commands, Cloud SDK commands, or custom tooling.
Triggers: Start builds from repository events, branch or tag patterns, or manual submissions.
Substitutions: Parameterize image names, regions, and versions. Built-in values such as COMMIT_SHA and BUILD_ID are available for supported triggers.
Service accounts: Control what the build can access. The build service account should have only the permissions required by the pipeline.
Caching: Reuse dependencies or build outputs to reduce execution time and cost.
Artifacts: Container images commonly go to Artifact Registry; build outputs and logs can also use Cloud Storage.
Promotion: Cloud Build can publish artifacts, while Cloud Deploy is generally preferable for controlled promotion through environments such as development, staging, and production.
Cloud Build Compared With AWS and Azure
Capability
Google Cloud
AWS
Azure
Managed build execution
Cloud Build
CodeBuild
Azure Pipelines
Primary configuration
cloudbuild.yaml
buildspec.yml
azure-pipelines.yml
Separate orchestration service
Cloud Deploy and related services
CodePipeline
Usually jobs within Azure Pipelines
Artifact container registry
Artifact Registry
Elastic Container Registry
Azure Container Registry
Cloud Build is usually a strong choice when the workload already runs on GCP and needs integration with IAM, Artifact Registry, Cloud Build triggers, and Google Cloud operational tooling. Azure Pipelines provides a broader workflow suite, while CodeBuild separates build execution from pipeline orchestration more explicitly.
Production Operations
Monitor builds: Alert on failure rate, queue delay, build duration, and infrastructure-related errors using Cloud Monitoring and Cloud Logging.
Control promotion: Do not deploy production directly from an untrusted pull request. Require protected branches, approval gates, tests, and provenance checks.
Manage cost: Cloud Build consumes compute resources according to the selected worker configuration. Reduce unnecessary parallelism, use appropriate machine types, and enable caching.
Handle capacity limits: Organization quotas can delay or reject builds. Monitor quota consumption and avoid scheduling large builds without capacity planning.
Retain evidence: Configure log retention according to audit and incident-response requirements.
Test pipeline changes: Treat builder-image updates and base-image changes like production changes by validating them in a non-production pipeline.
Security and Failure Modes
Use a least-privilege service account instead of broad project-wide roles.
Retrieve credentials from Secret Manager rather than storing them in source, substitutions, or image layers.
Avoid Docker socket mounting or privileged containers unless the pipeline has a documented, reviewed need.
Pin builder images and dependencies by immutable version or digest where practical.
Treat network failures, registry outages, and flaky tests as distinct incidents. Scripts should implement safe, idempotent retries because Cloud Build does not generally retry an arbitrary failed step.
Record the source commit, builder configuration, artifact digest, and deployment approval together so an artifact can be traced back to the code that produced it.
63
How does Google Cloud Artifact Registry work?
KnownRevise
Artifact Registry is Google Cloud’s managed repository for storing and distributing container images and software packages. It gives teams a private, IAM-controlled alternative to public registries such as Docker Hub while integrating directly with GKE, Cloud Run, Cloud Build, and other Google Cloud services.
Artifact Registry organizes artifacts into repositories scoped to a Google Cloud project and a location, such as us-central1 or europe-west1.
Container images: Docker and OCI images used by GKE, Cloud Run, and Compute Engine.
Language packages: Formats such as Maven, npm, Gradle, Python, and Go.
Versioned artifacts: Each artifact version is stored by digest, while tags provide movable names such as latest or v2.4.1.
Managed service: Google Cloud operates the registry, including storage, scaling, and service maintenance; customers do not run registry containers themselves.
The repository format determines which artifact types it accepts. For example, a Docker repository should not be used to store Maven artifacts.
How It Works
A typical container-image workflow is:
A CI pipeline builds an image and pushes it to a repository.
Artifact Registry authenticates the publisher and stores the image manifest and layers.
A tag such as v2.4.1 points to the immutable image digest.
GKE, Cloud Run, or another runtime authenticates and pulls the required manifest and layers.
The runtime starts the container using the resolved image.
Images in that repository are addressed using a hostname such as us-central1-docker.pkg.dev/example-project/app-images/image-name:tag.
Production Operations
Region selection: Repository location affects latency, data residency, and resilience. Choose the location close to the workloads that pull the artifacts.
Multi-region: Dual-region and multi-region repository locations can improve availability, but they introduce replication behavior and should be selected according to recovery requirements.
Monitoring: Track push failures, pull errors, latency, storage growth, and quota-related issues through Cloud Monitoring and Cloud Audit Logs.
Availability: A regional repository should not be treated as a cross-region backup. For critical release artifacts, replicate or promote digests to another repository and location.
Retention: Deleting a version does not merely remove a tag; it removes the referenced artifact. Define retention and cleanup rules that match release and rollback requirements.
Upgrades: The service is server-side and requires no customer-managed infrastructure upgrades, but consumers should still test new application and platform releases before production rollout.
Production deployments should prefer immutable digests over tags. Otherwise, moving a tag such as latest can silently cause different instances to run different code.
Security and Failure Modes
Use IAM to grant only the required repository permissions:
roles/artifactregistry.reader: Pull artifacts.
roles/artifactregistry.writer: Push and pull artifacts.
Check the principal and repository-level IAM binding
Pull timeouts
Network policy, VPC Service Controls, or regional service issue
Review network paths and Google Cloud service status
Unexpected image version
A mutable tag was moved
Deploy by digest and control who can push tags
Storage or quota pressure
Excess versions or insufficient quotas
Apply retention rules and review quotas and billing
For external or user identities, prefer Workload Identity Federation over long-lived service-account keys. Sensitive repositories should also use least-privilege IAM, audit logging, vulnerability scanning through services such as Container Analysis, and encryption controls appropriate to organizational policy.
AWS and Azure Comparison
Capability
Google Cloud
AWS
Azure
Managed artifact registry
Artifact Registry
Amazon ECR
Azure Container Registry
Container images
Supported
Supported
Supported
Language package hosting
Maven, npm, Gradle, Python, Go, and others
Primarily container images; CodeArtifact for packages
Primarily container images; Azure Artifacts for packages
Private IAM integration
Google Cloud IAM
AWS IAM
Azure RBAC and Entra ID
Common consumption
GKE and Cloud Run
EKS and ECS
AKS and Container Apps
Best fit
GCP-native CI/CD and container platforms
AWS-centric workloads
Azure-centric workloads
Choose Artifact Registry for private container and package distribution on Google Cloud, especially when CI/CD, GKE, and Cloud Run already use Google Cloud IAM and networking.
64
How does Google Cloud Deploy work?
KnownRevise
Google Cloud Deploy is a managed release-orchestration service for delivering containerized applications to GKE, Cloud Run, and Cloud Run jobs. It builds a versioned release once, then progressively promotes that same release through development, staging, and production targets.
Cloud Deploy coordinates the path from a source revision to a running application:
Cloud Build or another CI system builds and tests the application.
Skaffold defines how the application is built, packaged, and deployed.
Cloud Deploy creates a release and manages rollouts across one or more targets.
The target environment performs the actual deployment, such as updating a GKE workload or a Cloud Run service.
Cloud Deploy is not a complete CI/CD platform by itself. It focuses on controlled delivery, promotion, and rollout verification.
Core Concepts
Delivery pipeline: Defines the ordered sequence of targets through which a release moves.
Target: Represents an environment or deployment destination, such as a GKE cluster, Cloud Run service, or Cloud Run job.
Release: A versioned deployment unit containing the application image and deployment configuration.
Rollout: One attempt to deliver a release to a particular target.
Execution configuration: Controls target-specific behavior, including service accounts, permissions, GKE namespaces, Cloud Run services, rollout strategy, and approval requirements.
Rollout strategy: Supports standard deployment and canary delivery, where a configured percentage of traffic receives the new version before full rollout.
A release can be promoted without rebuilding it. This reduces differences between what passed testing and what reaches production.
Production Behavior and Failure Handling
Cloud Deploy supports manual promotion, approval gates, canary rollouts, and verification through the rollout strategy. Verification commonly relies on health checks and Cloud Monitoring signals.
Operational behavior is important:
A failed rollout is recorded against the release and target. It does not automatically prove that the previous production version remains healthy.
A failed canary can affect a subset of production traffic, so error rates, latency, saturation, and business metrics should be evaluated before further promotion.
Inspect release and rollout status using commands such as gcloud deploy releases describe and gcloud deploy rollouts describe.
Cloud Deploy does not replace database migration tooling, feature-flag management, or application-level rollback procedures.
Teams should define who can approve, retry, promote, and roll back a release before an incident occurs.
Cloud Logging and Cloud Monitoring should capture rollout events and correlate them with application telemetry. High-risk releases should use canaries, small traffic increments, explicit acceptance criteria, and a documented rollback decision owner.
Security and Operational Trade-offs
Use separate identities for pipeline execution, release creation, promotion, and runtime workloads. Grant only the permissions each target requires, and avoid embedding credentials in Skaffold files or release metadata.
For sensitive workloads:
Use private GKE clusters where appropriate.
Store application secrets in Secret Manager rather than manifests.
Use dedicated service accounts and avoid broad primitive roles.
Restrict promotion permissions to authorized release operators.
Review Cloud Audit Logs for deployment and approval activity.
Evaluate VPC Service Controls and customer-managed encryption keys for workloads with strict isolation or compliance requirements.
Cloud Deploy adds little direct infrastructure cost, but deployments still incur charges for Cloud Build, GKE or Cloud Run, logs, storage, and network egress.
AWS and Azure Equivalents
Capability
Google Cloud
AWS
Azure
Release orchestration
Cloud Deploy
CodePipeline with CodeDeploy
Azure DevOps Pipelines with Environments
Build and test
Cloud Build
CodeBuild
Azure Pipelines
Container deployment
GKE or Cloud Run
EKS or ECS
AKS or Azure Container Apps
Controlled promotion
Targets and rollouts
Deployment groups and actions
Environments and approvals
These products are not identical. AWS CodePipeline is a broader workflow orchestrator, while Azure DevOps provides pipeline and environment controls as part of a larger application-lifecycle platform.
Operational Best Practices
Keep pipeline, target, and release names consistent across regions and environments.
Promote immutable, uniquely tagged images rather than mutable tags.
Require automated tests and security scans before production promotion.
Use approval gates for sensitive targets and canary delivery for high-traffic services.
Monitor deployment health with service-level indicators, not only container readiness.
Test promotion and rollback procedures regularly under production-like conditions.
Record the release identifier in change tickets and incident timelines for traceability.
65
How should I manage GCP infrastructure with Terraform?
KnownRevise
Use Terraform’s Google provider to declare GCP resources as code, review the resulting plan, and apply changes through Google Cloud APIs. Store Terraform state in a private Google Cloud Storage bucket so teams share a consistent view of production infrastructure.
Run terraform init to install providers and configure the GCS backend.
Run terraform plan; by default, Terraform refreshes remote objects before calculating changes.
Review the plan for replacements, IAM changes, exposed resources, and quota impact.
Run terraform apply to execute the approved changes through GCP APIs.
Persist resource attributes in state for subsequent operations.
GCP’s hierarchy is normally organization → folders → projects. Permissions, organization policies, and service quotas can prevent otherwise valid plans from being applied.
Cloud Run for stateless HTTP services; GKE when Kubernetes is required; App Engine for its managed application platform; Cloud Functions for event-driven functions
Cloud SQL for relational workloads, Spanner for globally distributed transactions, Firestore for documents, Bigtable for wide-column data, and BigQuery for analytics
Remote state: Keep the GCS backend private, enable versioning and an appropriate retention or recovery policy, restrict access to the execution identity, and regularly test recovery.
Authentication: Use Application Default Credentials locally. In CI/CD, use Workload Identity Federation or workload identity rather than stored service-account keys.
Change control: Commit reviewed configuration and terraform.lock.hcl, require plan approval, save plans securely, and apply the exact reviewed plan.
Concurrency: Serialize pipelines that target the same state prefix because Terraform uses the backend lock.
Operations: Monitor availability, latency, errors, and saturation with Cloud Monitoring and Cloud Logging; use labels, budgets, and billing exports to control cost.
Failure Modes and Trade-offs
State loss: Losing state makes resource ownership difficult to reconstruct. Test restoration from a prior bucket version.
Partial apply: A resource may be created before Terraform records it in state. Find it by stable GCP name or ID and import it instead of creating a duplicate.
Replacement risks: Changes such as a bucket name or region can force replacement. Use prevent_destroy for critical resources and review destructive plans carefully.
API failures: IAM errors, organization policies, quota exhaustion, and transient API failures can stop an apply. Retry only operations known to be safe and verify final GCP state.
Backup limitations: Terraform manages infrastructure, not application data. Configure Cloud SQL backups, Firestore recovery, GKE volume backups or snapshots, and other service-specific recovery procedures.
Security and Operational Practices
Grant only the permissions required by each workflow; avoid broad roles such as roles/owner and roles/editor.
Treat state and saved plans as sensitive. Marking an output sensitive = true only hides it from CLI output; it does not remove the value from state.
Store secrets in Secret Manager and reference them from workloads instead of placing secret values directly in Terraform variables.
Use organization policies, VPC Service Controls where appropriate, and narrowly scoped IAM bindings to constrain production changes.
Upgrade providers in a separate change, review the provider changelog, commit the lock file, and validate upgrades in a non-production environment before promotion.
66
What is Vertex AI in Google Cloud?
KnownRevise
Vertex AI is Google Cloud’s unified machine learning platform for building, training, deploying, and operating ML models and generative AI applications. It combines managed training, online and batch prediction, model governance, workflow orchestration, and access to Google’s foundation models such as Gemini through a shared control plane, console, and set of service APIs.
Read full answerHide full answer
Applies toGoogle Cloud Vertex AI (current stable service)
What It Provides
A single control plane for the ML lifecycle: data preparation support, training, evaluation, deployment, monitoring, and governance.
Managed infrastructure for custom training jobs, including distributed training, hyperparameter tuning, and containerized workloads.
Managed serving through endpoints with autoscaling, traffic splitting, and separate batch prediction.
MLOps capabilities such as Vertex AI Pipelines, Model Registry, Vertex AI Experiments, and Model Monitoring.
Generative AI access through supported Vertex AI APIs, including Gemini models, embeddings, and tuning and evaluation capabilities.
Core Components
Component
Purpose
Training
Runs custom or AutoML training jobs on managed compute
Endpoints
Serves models for online prediction with scaling and versioning
Batch Prediction
Runs large-scale offline inference
Model Registry
Manages model versions, artifact references, and metadata
Pipelines
Orchestrates reproducible ML workflows
Feature Store
Manages and serves ML features consistently
Workbench
Provides managed notebook environments for development
Model Monitoring
Detects skew and drift for supported model types and analyzes prediction logs; prediction-quality analysis requires suitable ground-truth data and configuration
When to Use It
Use AutoML or managed training when you want Google Cloud to handle much of the model development and infrastructure.
Use custom training when you need control over frameworks, containers, distributed strategies, or specialized hardware such as GPUs and TPUs.
Use Vertex AI endpoints for low-latency online inference and batch prediction for large offline scoring jobs.
Use supported generative AI APIs for workloads that need enterprise controls such as IAM, VPC Service Controls, audit logging, and—where offered—regional data residency.
Operational Considerations
Cost is driven by training compute, endpoint minimum replicas, accelerator usage, storage, and prediction traffic. Choose minimum replica counts deliberately; scale-to-zero configurations can reduce cost but must satisfy cold-start and availability requirements.
Not every Vertex AI feature is available in every region. Validate service, model, and feature availability before designing multi-region or regulated workloads.
Secure access with least-privilege IAM, appropriately scoped service accounts, VPC Service Controls, and Cloud Audit Logs. Apply Private Service Connect or customer-managed encryption keys only where supported by the specific service and region.
Monitor training jobs, endpoint latency, error rates, quota usage, and model drift; configure alerts before production traffic depends on an endpoint.
Plan model versioning, rollback, and traffic splitting so failed deployments can be reversed without downtime.
Comparison with Other Clouds
Capability
Google Cloud
AWS
Azure
Managed ML platform
Vertex AI
Amazon SageMaker
Azure Machine Learning
Managed generative AI models
Gemini via Vertex AI
Amazon Bedrock
Azure OpenAI Service
ML pipelines
Vertex AI Pipelines
SageMaker Pipelines
Azure Machine Learning Pipelines
Online model serving
Vertex AI Endpoints
SageMaker Endpoints
Azure ML managed endpoints
Best Practices
Keep training and serving containers reproducible and versioned.
Register production model versions in Model Registry and retain evaluation evidence and lineage in Vertex AI Model Registry or Vertex ML Metadata as appropriate.
Separate development, staging, and production projects or environments.
Use pipelines for repeatable retraining rather than manual notebook execution.
Test endpoint scaling, rollback, and monitoring before exposing a model to production traffic.
67
How can a machine learning model be deployed on Vertex AI?
KnownRevise
Deploying a model on Vertex AI means registering a trained model artifact in the Model Registry, creating an Endpoint, and deploying the model to that endpoint with a chosen machine type and autoscaling policy. Vertex AI then serves online predictions over HTTPS and can also run batch prediction jobs against Cloud Storage or BigQuery data.
Read full answerHide full answer
Applies toGoogle Cloud Vertex AI (current stable)
How It Works
The deployment flow has three stages:
Upload the model to the Model Registry, pointing at a container image (pre-built or custom) and the artifact location in Cloud Storage.
Create an endpoint, which is the serving resource that exposes a prediction URL.
Deploy the model to the endpoint, specifying machine type, replica counts, and traffic split.
Vertex AI handles load balancing, autoscaling, health checks, and logging to Cloud Monitoring automatically. A single endpoint can host multiple model versions with traffic splitting for canary rollouts.
Deployment Options
Approach
When to use
Container
Pre-built prediction container
Standard frameworks (scikit-learn, XGBoost, TensorFlow, PyTorch)
Google-provided image
Custom container
Custom inference code, preprocessing, or unsupported frameworks
Your own image in Artifact Registry
Batch prediction
Offline scoring of large datasets
No endpoint needed
Online prediction suits low-latency, request/response workloads. Batch prediction is cheaper for bulk scoring and writes results to Cloud Storage or BigQuery.
Key Configuration
Machine type: choose based on model size and latency needs (e.g., n1-standard-4, n1-standard-8, or accelerator-backed types with GPUs).
Autoscaling: set --min-replica-count and --max-replica-count; Vertex AI scales between them based on request load.
Traffic split: deploy a new version with a percentage of traffic for safe rollouts, then promote to 100%.
Explainability: enable Vertex AI Explainable AI for feature attributions when supported by the framework.
Monitoring: model monitoring detects training-serving skew and prediction drift; alerts route through Cloud Monitoring.
Comparison with Other Clouds
Capability
Vertex AI
AWS SageMaker
Azure ML
Managed endpoint
Vertex AI Endpoint
SageMaker Endpoint
Azure ML Managed Endpoint
Model registry
Vertex AI Model Registry
SageMaker Model Registry
Azure ML Model Registry
Batch scoring
Batch prediction job
Batch Transform
Batch endpoint
Autoscaling
Built-in replica scaling
Auto Scaling policies
Managed endpoint scaling
Code Example
# 1. Upload model to the registry
gcloud ai models upload \
--region=us-central1 \
--display-name=my-sklearn-model \
--container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest \
--artifact-uri=gs://my-bucket/models/sklearn-model/
# 2. Create an endpoint
gcloud ai endpoints create \
--region=us-central1 \
--display-name=my-prediction-endpoint
# 3. Deploy the model to the endpoint (replace IDs from previous output)
gcloud ai endpoints deploy-model ENDPOINT_ID \
--region=us-central1 \
--model=MODEL_ID \
--display-name=sklearn-v1 \
--machine-type=n1-standard-4 \
--min-replica-count=1 \
--max-replica-count=4 \
--traffic-split=0=100
# 4. Send an online prediction request
gcloud ai endpoints predict ENDPOINT_ID \
--region=us-central1 \
--json-request=request.json
Use a dedicated service account for the endpoint with least-privilege access to Cloud Storage artifacts; avoid the default compute service account in production.
Pin container image digests rather than mutable tags to make deployments reproducible.
Set --min-replica-count=1 for latency-sensitive services to avoid cold starts; scale to zero only for non-critical workloads.
Enable model monitoring from day one to catch drift; define alerting thresholds before launch.
Test with a traffic split before full promotion; roll back by redeploying the previous version at 100%.
Watch costs: accelerator-backed machines and always-on replicas are the main cost drivers; right-size and use batch prediction where latency allows.
Common pitfall: deploying a model whose container image and artifact format mismatch (e.g., scikit-learn version skew) causes runtime load failures; validate locally with the same container first.
68
How does GCP billing work and how do you set budget alerts?
KnownRevise
GCP billing centers on a billing account that pays for one or more projects and meters usage by SKU. Budgets are alarm thresholds rather than hard spending caps, and can notify operators or publish to Pub/Sub for automated response.
Read full answerHide full answer
Applies toGoogle Cloud current stable, Google Cloud CLI current stable, and Terraform Google Provider 7.x
How GCP Billing Works
A billing account is the payment entity. It can pay for multiple projects, while each project has only one billing account at a time.
Usage is metered per SKU and then priced according to service, region, resource type, pricing model, and applicable discounts. Eligible credits—such as free-trial, promotional, spend-based, commitment, or negotiated credits—are applied automatically to the resulting charges.
Compute Engine may receive Sustained Use Discounts, while eligible Compute Engine, memory, and related usage may receive Committed Use Discounts for one- or three-year commitments.
The console provides billing reports and invoice views. BigQuery billing export is useful for repeatable SQL analysis, but it is not authoritative: export data can be delayed or differ from the invoice because of credits, adjustments, and late-arriving usage, so finance reconciliation must use the invoice.
Labels are useful for allocating costs, but not every GCP resource supports labels. Keep their keys consistent and include project, environment, service, and owner where applicable.
Budgets and Alerts
A budget has an amount, scope, and threshold rules. It can cover a billing account or selected projects and can filter eligible services, labels, annotations, and other billing dimensions.
Thresholds use actual or forecast spend. Forecast thresholds depend on sufficient usage history and Cloud Billing forecast data, so actual-spend thresholds are a safer minimum.
Notifications can go to email recipients or a Pub/Sub topic. Pub/Sub only publishes an event; a separate subscriber must implement actions such as scaling down non-production capacity.
A budget does not stop consumption or create a spending ceiling. Quotas can constrain particular resources or request rates, but they are not a general spend cap.
Cloud Billing must have permission to publish to the selected topic. Allow for IAM propagation and monitor both the budget service and the subscriber.
Cost Management and Comparison
Aspect
GCP
AWS
Azure
Detailed cost export
BigQuery
Cost and Usage Report in S3, queried with Athena
Cost Management exports
Budget behavior
Alarm and notification only
Alarm and notification only
Alarm and notification only
Universal spend cap
None
None
None
Usage guardrails
Quotas and organization policies
Service quotas
Quotas and Azure Policy
Discounts
SUDs and CUDs
Savings Plans and Reserved Instances
Savings Plans and Reservations
Use the Recommender to identify idle VMs, unattached disks, and rightsizing opportunities.
Use CUDs for stable eligible workloads and Spot VMs for interruptible batch processing.
Use quotas and organization policies as resource guardrails, but monitor billable services such as egress, which may not have a meaningful quota.
Enable BigQuery billing export early, retain the export configuration in Terraform, and reconcile anomalies against the invoice.
Enforce label conventions through provisioning policy and CI checks; use Resource Manager tag policies when managing supported tags.
Treat budgets as alarms. Use quotas to limit eligible usage and make any billing-disablement or scaling automation idempotent, tested, and protected by approvals.
Give the Pub/Sub consumer only the permissions required for its response, and monitor undelivered messages, failed alerts, and forecast accuracy.
Review CUD commitments and Spot VM interruption settings regularly.
Track egress, cross-region traffic, data processing, and license charges, which are common sources of unexpected cost.
Separate unrelated business units into billing accounts when chargeback, access control, or ownership differs.
69
What are Google Cloud committed use and sustained use discounts?
KnownRevise
Committed Use Discounts (CUDs) are contractual discounts for predictable consumption of eligible Google Cloud resources over a one- or three-year term. Sustained Use Discounts (SUDs) are automatic discounts for eligible Compute Engine resources that remain running for a substantial portion of the billing month.
Read full answerHide full answer
Applies toGoogle Cloud Platform (current stable billing and pricing model; eligibility and rates vary)
Core Difference
Characteristic
Committed Use Discount
Sustained Use Discount
Purchase model
You purchase a commitment
No purchase is required
Commitment period
One or three years
None
Eligible resources
Eligible Compute Engine resources and other supported services, depending on commitment type
Certain Compute Engine machine types in eligible regions
Discount trigger
Usage is covered by an active commitment
Eligible resources run continuously for a sufficient portion of the month
Payment risk
You pay for the commitment even if usage is lower than expected
No committed spend; stopping usage reduces the discount
Capacity guarantee
Generally none
None
How CUDs Work
CUDs are purchased through a Google Cloud billing account and commonly take these forms:
Resource-based CUDs: Apply to specific Compute Engine resources, such as vCPU, memory, GPUs, or local SSD, and usually have regional scope.
Spend-based CUDs: Provide discounts against eligible spend across supported services without requiring the same resource-level configuration in every project.
The typical workflow is:
Identify a stable baseline workload and its billing account.
Select the eligible resource or spend scope, region where applicable, commitment duration, and discount level.
Purchase the commitment.
Match the commitment against eligible usage in associated projects.
Monitor utilization and remaining commitment before the commitment expires.
A commitment is primarily a billing commitment, not a capacity reservation. It does not normally guarantee that Compute Engine capacity will be available. Quotas, reservations, and regional capacity must be managed separately.
How SUDs Work
SUDs apply automatically to eligible Compute Engine usage. There is no contract or prepayment, but the discount depends on how continuously the resource runs during the billing month.
For example, a long-lived internal VM that runs throughout the month can receive a SUD. If the VM is repeatedly stopped and started, the accumulated usage pattern may not qualify for the same discount. This makes SUDs most useful for stable services, development environments that remain online, and autoscaled fleets whose underlying nodes remain active.
SUDs should not be assumed to apply to every Compute Engine product. Spot VMs, preemptible workloads, some machine families, and some managed services may not qualify.
AWS and Azure Equivalents
Google Cloud
AWS
Azure
CUD
EC2 Reserved Instances or Savings Plans
Reservations or Savings Plans
SUD
No broadly equivalent automatic VM-use discount
No broadly equivalent automatic VM-use discount
AWS and Azure generally achieve sustained-use savings through reservations, savings plans, or product-specific discount programs rather than a directly equivalent automatic mechanism.
Production Considerations
Budget monitoring: Configure budgets and alerts in Cloud Billing, and export detailed billing data to BigQuery for commitment-utilization analysis.
Underutilization: A CUD can become expensive if workloads are migrated, deleted, or resized. Compare the discount against realistic steady-state consumption before purchasing.
Expiry planning: Track commitment end dates. If a commitment is not renewed, covered usage can return to standard pricing.
Regional scope: Resource-based commitments may not cover the same workload after migration to another region.
Capacity and quotas: A discount does not remove quota limits or guarantee regional capacity.
Security and governance: Restrict commitment purchases with appropriate billing-account IAM permissions. Financial discounts do not replace workload identity, least-privilege access, audit logging, or data-protection controls.
Workload selection: Use CUDs for stable, committed baselines and planned multi-year capacity. Use SUDs for continuously running eligible VMs, and consider Spot VMs for fault-tolerant batch work where interruption is acceptable.
For billing-account administration, gcloud billing accounts list and gcloud billing accounts describe BILLING_ACCOUNT_ID help identify the account that owns the commitment and associated billing scope.
70
How can you reduce Google Cloud Platform costs?
KnownRevise
The most effective GCP cost reduction combines rightsizing, removing idle resources, matching pricing and capacity to demand, and continuously controlling spend. Use cost controls that preserve reliability, security, and recovery objectives rather than optimizing the invoice in isolation.
Read full answerHide full answer
Applies toGoogle Cloud current stable services, Google Cloud CLI current stable release, and BigQuery CLI current stable release
Start with Cost Attribution
Before changing capacity, attribute spending and ownership:
Resource hierarchy and IAM: Use folders and projects to separate environments and ownership boundaries, and grant least privilege at the narrowest practical scope.
Labels and billing export: Label supported resources and export detailed charges to BigQuery; analyze spend by project, service, SKU, label, and day.
Idle resources: Review idle VMs, disks, IP addresses, snapshots, Cloud SQL instances, GKE node pools, and database capacity.
Shared costs: Attribute shared networking and logging to a platform project or cost center so teams are not discouraged from using required controls.
Match Capacity and Pricing to Demand
Lever
Best use
Operational trade-off
Autoscaling
GKE, Cloud Run, Cloud Functions, and App Engine
Requires useful metrics, safe scale-out latency, and maximum-capacity limits
Spot VMs
Batch, CI, and fault-tolerant jobs
Workloads must checkpoint and retry after preemption
Committed use discounts
Predictable Compute Engine and eligible GKE node usage
Commitments must match resource and region; unused commitments still incur charges
Rightsizing
CPU, memory, disks, and database capacity
Insufficient headroom can increase latency or cause outages
Scale-to-zero
Cloud Run, Cloud Functions, and App Engine standard
Cold starts affect latency
Prefer flexible machine families and current-generation hardware when benchmarks show better price/performance. Test migrations against representative workloads rather than assuming a newer instance type is cheaper.
Control Data, Service, and Network Costs
BigQuery: Partition large tables by date, cluster by frequently filtered columns, and avoid SELECT *. Use --maximum_bytes_billed to reject unexpectedly large queries. Compare total on-demand cost before buying slot reservations.
Cloud Storage: Use lifecycle rules to delete temporary data and transition infrequently accessed objects to Nearline, Coldline, or Archive. Include noncurrent object versions.
Cloud SQL: Stop nonproduction instances, remove replicas not required by recovery objectives, and right-size CPU, memory, and storage. Backups and retained logs also consume storage.
GKE: Enable node autoscaling, remove permanently idle pools, and monitor managed-cluster fees, load balancers, and persistent disks.
Cloud Run, Cloud Functions, and App Engine: Use zero-minimum instances where supported, cap maximum instances, tune concurrency, and select suitable instance classes. App Engine flexible always keeps at least one instance running, so it cannot scale to zero.
Spanner and Bigtable: Right-size Spanner nodes and replicas; review Bigtable cluster nodes, replication, and retained backups. Both may continue charging after traffic falls.
Firestore and Pub/Sub: Minimize document operations, retry amplification, subscriber fan-out, and duplicate subscriptions. Use TTL for expired Firestore documents and account for Pub/Sub deliveries and snapshot storage.
VPC networking and load balancing: Review egress, NAT, inter-region traffic, and load-balancer footprint. Use regional or internal load balancing where architectural requirements permit.
Logging and monitoring: Exclude unnecessary verbose logs, bound retention, and control metric cardinality. Preserve security and incident-response evidence.
Budgets are alerts, not automatic spending caps. Combine them with quotas, least-privilege IAM, and organization policies to constrain runaway workloads.
Production Guardrails
A production cost program should include:
Daily billing analysis and anomaly detection.
Budget alerts with explicitly configured recipients.
Ownership labels enforced through infrastructure as code.
Automated shutdown for explicitly nonproduction environments.
Gradual rollout, SLO comparison, and rollback procedures for rightsizing and upgrades.
Protect required backups, snapshots, replicas, and recovery processes; lower spending must not weaken service objectives.
Code Example
These commands inspect labels, create an autoscaled zonal GKE node pool, create budget thresholds, and limit billable BigQuery bytes:
# Show Compute Engine instances and their attribution labels.
gcloud compute instances list \
--project=PROJECT_ID \
--format="table(name,zone,machineType,status,labels)"
# Add or update labels without removing unrelated existing labels.
gcloud compute instances update INSTANCE_NAME \
--project=PROJECT_ID \
--zone=ZONE \
--update-labels=env=prod,team=platform,cost-center=123
# Start with one node and scale according to workload demand.
gcloud container node-pools create platform-pool \
--project=PROJECT_ID \
--cluster=CLUSTER_NAME \
--zone=ZONE \
--machine-type=e2-standard-4 \
--num-nodes=1 \
--enable-autoscaling \
--min-nodes=1 \
--max-nodes=10
# Create budget thresholds at 50%, 90%, and 100%.
gcloud billing budgets create \
--billing-account=BILLING_ACCOUNT_ID \
--display-name=platform-budget \
--budget-amount=1000USD \
--threshold-rule=percent=0.5 \
--threshold-rule=percent=0.9 \
--threshold-rule=percent=1.0
# Fail rather than bill more than 1 GB of scanned data.
bq query \
--project=example-project \
--use_legacy_sql=false \
--maximum_bytes_billed=1000000000 \
'SELECT COUNT(*) FROM `example-project.example_dataset.events`'
Best Practices and Common Pitfalls
Optimize total cost: A cheaper configuration that causes throttling or retries may increase overall cost.
Model commitments carefully: Include growth, shutdowns, migrations, and unused-commitment charges.
Automate recurring waste: Prefer lifecycle policies, scheduled shutdowns, and autoscaling over manual cleanup.
Preserve recovery capabilities: Do not remove backups, replicas, or logging required by recovery objectives.
Validate every saving: Compare cost, latency, error, and saturation metrics before and after each change.
71
How do GCP services compare to AWS equivalents?
KnownRevise
GCP and AWS offer largely parallel cloud platforms, but their regional defaults, IAM structures, billing models, and operational trade-offs differ enough that services should be mapped by capability rather than by name alone.
Read full answerHide full answer
Applies toGoogle Cloud Platform and Amazon Web Services current stable service catalogs (2025)
Core Service Mapping
Category
GCP
AWS Equivalent
Operational distinction
Virtual machines
Compute Engine
EC2
Compute Engine meters per second with a one-minute minimum and offers eligible sustained-use discounts; current-generation EC2 also meters per second and supports On-Demand, Savings Plans, Reserved Instances, and Spot.
Managed Kubernetes
GKE Standard / Autopilot
EKS / Fargate
Autopilot provides an opinionated managed-node configuration; Fargate supplies serverless node capacity for EKS.
Serverless containers
Cloud Run
App Runner / Fargate
Cloud Run can scale to zero when minimum instances are zero; it bills for requests and allocated CPU/memory time, rounding each request’s billable time up to the nearest 100 ms.
Managed application PaaS
App Engine
Elastic Beanstalk / App Runner
App Engine standard manages deployment and scaling closely; Beanstalk gives more infrastructure control, while App Runner is comparatively simple.
Serverless functions
Cloud Functions, 2nd gen
Lambda
2nd-gen Cloud Functions are built on Cloud Run infrastructure.
Object storage
Cloud Storage
S3
Both use globally unique bucket names. An S3 bucket has one home Region, and its objects are not globally replicated by default.
Relational databases
Cloud SQL
RDS / Aurora
Both support managed MySQL and PostgreSQL deployments, private connectivity, backups, high availability, and automated maintenance.
Globally distributed SQL
Spanner
Aurora DSQL / Aurora Global Database
Aurora DSQL is the closest current AWS analogue, but its PostgreSQL interface and architecture differ from Spanner.
Document NoSQL
Firestore
DynamoDB
Firestore databases default to a multi-region location unless a regional location is selected; DynamoDB multi-region replication uses Global Tables.
Wide-column NoSQL
Bigtable
Keyspaces / DynamoDB use cases
Bigtable supports large operational and analytical workloads; Keyspaces is Cassandra-compatible, while DynamoDB has a different data model.
Data warehouse
BigQuery
Redshift / Athena
BigQuery is serverless and separates storage from query-compute charges.
Messaging and streaming
Pub/Sub
SNS + SQS / Kinesis
Pub/Sub combines topic distribution with pull/push subscriptions; AWS separates notification, queueing, and streaming into distinct services.
Networking
VPC
VPC
A GCP VPC is a global resource with regional subnets; an AWS VPC is regional.
Load balancing
Cloud Load Balancing
ALB / NLB
A GCP global external Application Load Balancer uses anycast frontend IPs; an AWS ALB is regional.
Monitoring
Cloud Monitoring + Cloud Logging + Cloud Trace
CloudWatch + X-Ray
CloudWatch covers metrics and logs; on GCP, metrics come from Cloud Monitoring and logs from the separate Cloud Logging service. Tracing is provided by Cloud Trace and X-Ray.
Secrets and keys
Secret Manager + Cloud KMS
Secrets Manager / Parameter Store + KMS
All support centrally managed secrets and envelope encryption; use IAM, rotation, audit logs, and customer-managed keys where required.
Key Production Differences
Resource hierarchy: GCP IAM policies can inherit through organizations, folders, and projects. AWS identity policies remain account-scoped, while Organizations, organizational units, SCPs, and StackSets provide multi-account governance.
Global versus regional defaults: GCP networks span regions, but subnets, Cloud NAT instances, and many regional backends remain regional. AWS VPCs and load balancers are regional unless separate global services are used.
Billing: GCP sustained-use discounts apply automatically only to eligible machine types; AWS commitment discounts generally require Savings Plans or Reserved Instances. Both also support discounted interruptible capacity through Spot Instances.
Workload identity: GCP Workload Identity Federation issues short-lived credentials for service accounts through federated external identity providers. AWS uses IAM roles and OIDC or SAML identity providers.
Network egress: Both charge for internet egress and cross-region or cross-zone transfers in many configurations. Model internal Google traffic, AWS service endpoints, private-service access, and NAT paths before estimating cost.
Control versus convenience: GCP generally offers opinionated managed modes such as GKE Autopilot and Cloud Run. AWS often provides more control at the infrastructure or orchestration layer, but that can increase node, networking, and upgrade work.
When to Choose Each
Choose GCP for BigQuery analytics, Spanner’s globally distributed strongly consistent SQL, a global VPC resource, or GKE Autopilot’s reduced node-management burden.
Choose AWS when a required AWS service, regional footprint, compliance commitment, or procurement relationship makes it the better operational fit.
For hybrid or multi-cloud: use Terraform, Kubernetes, PostgreSQL, and Kafka where practical, while retaining explicit exit plans for proprietary data and messaging APIs.
Operational Trade-offs
Observability: Standardize collection on OpenTelemetry, but configure metric, log, and trace retention and sampling explicitly rather than relying on provider-specific defaults.
Disaster recovery: Regional Cloud SQL deployments and regional GKE clusters can span zones; RDS Multi-AZ and EC2 Auto Scaling likewise use multiple Availability Zones. Multi-region recovery still requires an explicit, tested design.
Cost management: Use GCP Recommender, Cloud Billing budgets, and AWS Cost Explorer, Cost Optimization Hub, and AWS Budgets. Labels and tags are required for useful allocation and chargeback.
Backups: Managed backups do not replace restoration tests. Validate point-in-time recovery, backup retention, cross-region copies, and actual recovery-time and recovery-point objectives.
Best Practices
Compare service capabilities, SLAs, quotas, networking behavior, and operational burden before selecting an equivalent.
Apply least privilege with groups or service accounts, use short-lived credentials, and avoid long-lived access keys.
Manage infrastructure as code, isolate Terraform state by environment and cloud boundary, and review changes through version control.
Test failover, regional evacuation, restore procedures, quota exhaustion, and incident runbooks before production launch.
72
How should you architect high availability across GCP zones and regions?
KnownRevise
Use a layered high-availability (HA) design that removes single points of failure at each failure domain: spread replicas across zones, isolate regional stacks, and introduce global traffic routing where required. In GCP, multi-zone availability protects against a zone failure, while multi-region availability protects against a broader regional outage; neither alone provides complete disaster recovery.
Zone: Place replicas in different zones within a region. A zone can fail because of hardware, power, or a deployment issue while the rest of the region remains available.
Region: Deploy independent regional stacks when an entire region must be survivable. A regional outage can affect control planes, network paths, and regional managed-service endpoints simultaneously.
Global layer: Use a global external Application Load Balancer for stateless HTTP(S) workloads that must survive regional failures. For GKE or Compute Engine backends, the load balancer uses health checks to route traffic only to healthy endpoints.
Resource hierarchy: Organize resources under the organization, folders, and projects according to environment, ownership, and lifecycle. Apply IAM and policy controls at the highest practical level; use labels for ownership, cost, and regional deployment metadata.
Service Selection
Architecture pattern
GCP
AWS equivalent
Azure equivalent
Global HTTP(S) ingress
Global external Application Load Balancer
CloudFront with ALB
Front Door
Zone-resilient containers
Regional GKE cluster
Multi-AZ EKS
Zone-spread AKS
Regional managed relational database
Cloud SQL regional HA
RDS Multi-AZ
Zone-redundant Azure SQL
Multi-region transactional data
Spanner multi-region with global writes
Aurora Global Database with a regional writer
Cosmos DB with multi-region writes
Globally replicated document data
Firestore multi-region
DynamoDB global tables
Cosmos DB with multi-region replication
Durable object storage
Cloud Storage dual-region or multi-region
S3 with cross-region replication
Storage GRS/RA-GRS with paired-region replication
Choose Cloud Run for independently deployed, event-driven services; choose GKE when teams need Kubernetes APIs, custom scheduling, or platform extensibility. Use Cloud Storage for objects, Cloud SQL for relational transactions, Spanner for globally distributed transactions, and BigQuery multi-region for analytics—not low-latency transactional writes.
How the Architecture Behaves
A regional GKE cluster runs its control plane across three zones in that region. It survives individual zone failures but does not survive loss of the entire region.
Cloud SQL regional HA maintains a standby in another zone and can perform automatic failover. It does not provide active regional failover; cross-region read replicas are not a substitute for a tested write-failover design.
Cloud Run services deployed in multiple regions provide independent regional capacity. A global load balancer can route requests across regional serverless NEGs, although deployment and data-consistency policies must be defined separately.
Spanner multi-region replicates data across regions and coordinates writes globally. Its consistency, latency, and cost trade-offs differ from independently replicated databases.
Failure Modes and Trade-offs
Bad release: A deployment can break every healthy replica. Canary one region first, monitor error rates and latency, and retain a rapid rollback path.
Data-region outage: Stateless capacity can fail over, but a regional database may still block writes. Replicate data, define a promotion process, and document the accepted RPO and RTO.
Retry storm: A dependency outage can overwhelm recovering backends. Use bounded exponential backoff, jitter, circuit breakers, and idempotent operations.
Partial failure: A dependency may become slow rather than unavailable. Load-balancer health checks and timeouts must reflect user-visible symptoms.
Cost and latency: Duplicate regional capacity and cross-region traffic add cost. Evaluate data residency, egress charges, and whether every request truly requires global routing.
Security and Operational Controls
Use per-service service accounts, least-privilege IAM, Secret Manager, VPC Service Controls, and private connectivity where supported.
Restrict Cloud Run ingress and grant invocation access only to the required identity; avoid allUsers unless public access is intentional.
Use Admin Activity audit logs, which are always enabled, enable Data Access audit logs where required, and use Cloud Logging, Monitoring, Cloud Trace, and Error Reporting.
Protect databases with backups and point-in-time recovery where supported. Store recovery artifacts separately and perform scheduled restore tests.
Apply budgets, anomaly detection, and cost labels to duplicated regional infrastructure and inter-region transfer.
Production Checklist
Define failure scenarios and explicit RPO, RTO, and error-budget targets.
Verify capacity for losing an entire zone or region.
Exercise regional failover and database restoration regularly.
Confirm that retries are bounded and write operations are idempotent.
Monitor regional saturation, replication lag, failed requests, and failover events.
How do you migrate on-premises workloads to Google Cloud?
KnownRevise
A successful on-premises migration uses a migration wave: discover and classify the workload, build a secure landing zone, transfer data, validate it, and then cut over traffic with a tested rollback path. The appropriate GCP service depends on whether the workload is stateless, containerized, relational, distributed, or event-driven.
Read full answerHide full answer
Applies toGoogle Cloud CLI current stable and current Google Cloud managed services
Migration Strategy
Start with discovery rather than immediately copying servers:
Inventory: Record CPU, memory, disks, operating systems, licenses, dependencies, data stores, traffic patterns, and recovery objectives.
Classify: Mark each application as rehost, replatform, refactor, retire, or retain.
Wave dependencies: Migrate shared identity, DNS, network connectivity, databases, and secrets before dependent applications.
Success criteria: Set measurable targets for downtime, data loss, latency, error rate, and recovery time.
Pilot: Build a nonproduction migration and automate deployment before scaling across production waves.
Use separate projects or folders for landing, migration tooling, shared networking, and production. Centralize audit logs and billing, but grant access through groups and narrowly scoped IAM roles.
Choosing the Target Service
Workload
Recommended GCP service
Key consideration
Existing VM
Compute Engine
Fastest rehost path; guest customization and licensing still apply
Containerized application
GKE
Managed control plane, autoscaling, and workload portability
Stateless HTTP container
Cloud Run
Automatic scaling and revision-based deployments
Event-driven function
Cloud Functions
Trigger-driven execution without server management
Relational database
Cloud SQL
Managed PostgreSQL, MySQL, or SQL Server
Globally distributed transactional data
Spanner
Strong consistency and horizontal scale across regions
Application key-value state
Firestore or Bigtable
Firestore suits document state; Bigtable suits large operational workloads
Analytical data
BigQuery
Serverless SQL analytics and federated data exploration
Object data
Cloud Storage
Durable storage with lifecycle, versioning, and retention controls
VMware-heavy environment
VMware Engine
Extends the VMware operating model into GCP
App Engine remains appropriate for applications built around its platform model, but Cloud Run generally offers a more container-focused operational model for new HTTP services.
Connectivity and Data Transfer
Requirement
Approach
Moderate initial traffic or temporary hybrid setup
HA VPN with Cloud Router and BGP
Sustained high throughput or low-latency hybrid access
Cloud Interconnect, typically with a redundant topology
Large file transfers
Storage Transfer Service with on-premises agents
Supported relational databases
Database Migration Service for Cloud SQL targets
Incremental database synchronization
Database Migration Service or carefully tested change-data-capture tooling
Maintain dual-running paths until validation completes. Connectivity alone does not synchronize application data, and successful ping tests do not prove application correctness. Encrypt transit, restrict routes, monitor VPN or Interconnect health, and document the fallback when the private link fails.
Code Example
The following creates a minimal private network, imports a prepared VMware virtual disk, and starts a Compute Engine instance without a public IP.
Use IAP tunneling for administrative access rather than public SSH. Ensure the subnet has controlled private egress, such as Cloud NAT, when the application requires outbound Internet access.
For GKE, deploy the migrated container through a reviewed Kubernetes manifest rather than copying VM-specific configuration:
The application must expose /readyz and /healthz on port 8080. Configure GKE node autoscaling and disruption budgets for production availability.
Cutover and Validation
Reduce DNS TTL and confirm every dependency is available in GCP.
Perform the final database synchronization while writes are controlled.
Compare checksums, row counts, transaction outcomes, and business-level reports.
Run performance, failover, security, and restore tests.
Shift a small percentage of traffic, observe metrics, and increase gradually.
Retain the on-premises environment and a rollback plan through the agreed stabilization period.
Use Cloud Monitoring uptime checks, latency and error dashboards, and alerting on logs and metrics. If cutover fails, route traffic back, preserve diagnostic evidence, and reconcile any writes that occurred only on GCP.
Production Readiness
Security: Use IAM groups, service accounts, Secret Manager, VPC Service Controls, private service access, and least-privilege roles. Avoid long-lived service account keys.
Backups: Configure Compute Engine snapshots, Cloud SQL automated backups and point-in-time recovery, and tested GCS versioning or retention policies.
Upgrades: Use GKE release channels, Cloud Run revision rollouts, and staged database maintenance windows.
Cost: Apply labels, budgets, quota monitoring, rightsizing, and committed-use discounts where justified. Include network egress and idle provisioned capacity in migration estimates.
Common failure: Do not migrate databases independently of their applications; incompatible time zones, DNS behavior, certificates, firewall rules, and vendor licensing are frequent causes of late cutover failures.
74
How do you connect a Cloud Run service to Cloud SQL securely?
KnownRevise
Connect Cloud Run to Cloud SQL over the VPC network using Cloud SQL’s private IP, and authenticate with a least-privilege service account (either via the Cloud SQL Auth Proxy sidecar with IAM authentication, or via a password pulled from Secret Manager). Never expose the database on a public IP, and never bake credentials into the image.
Read full answerHide full answer
Applies toCloud Run (fully managed, second-generation runtimes) with Cloud SQL for PostgreSQL 16 / MySQL 8.0 (current GCP)
How It Works
Cloud Run is serverless, so “connecting to a database” means three things must line up:
Network path: Cloud Run must be able to reach the Cloud SQL private IP. Outbound traffic from Cloud Run to a VPC requires either a Serverless VPC Access connector or direct VPC egress; with either one, the PRIVATE_RANGES egress setting sends only private-address traffic into the VPC, while ALL_TRAFFIC sends everything. Without one, Cloud Run only has public internet egress and cannot reach a private-IP instance.
TLS: The Auth Proxy always encrypts its connection, but a direct private-IP connection is unencrypted unless the instance enforces SSL (--ssl-mode=ENCRYPTED_ONLY) and the client uses it. PostgreSQL clients should use sslmode=verify-full (with sslrootcert from the server CA) rather than disable.
Identity: the Cloud Run service account authenticates. Grant roles/cloudsql.client so the Cloud SQL Auth Proxy or connector can connect, plus roles/cloudsql.instanceUser only if the service account logs in with IAM database authentication.
Option
Connectivity
Credentials
Notes
Cloud SQL Auth Proxy sidecar
127.0.0.1:5432
IAM (--auto-iam-authn) or password
TLS terminated at the proxy; proxy is not a connection pooler
Direct private-IP connection
10.10.x.x:5432
Password from Secret Manager
Fewest moving parts; you handle TLS verification yourself
Public IP
Internet
IAM proxy or password
Avoid unless required; larger attack surface
Cloud Equivalents
GCP
AWS
Azure
Cloud Run
Lambda / ECS Fargate
Azure Container Apps
Cloud SQL for PostgreSQL
RDS for PostgreSQL
Azure Database for PostgreSQL
Secret Manager
Secrets Manager
Key Vault
Serverless VPC Access / direct VPC egress
Lambda VPC configuration / ECS awsvpc networking
VNet integration
Wiring It Up (gcloud)
export PROJECT_ID=example-project
export REGION=us-central1
gcloud config set project "$PROJECT_ID"
# 1. VPC with a subnet for Cloud Run and a range reserved for Cloud SQL
gcloud compute networks create app-vpc --subnet-mode=custom
gcloud compute networks subnets create app-subnet \
--network=app-vpc --region=$REGION --range=10.10.0.0/24 \
--enable-private-ip-google-access
gcloud compute addresses create cloudsql-range --global \
--purpose=VPC_PEERING --prefix-length=16 --network=app-vpc
gcloud services vpc-peerings connect --service=servicenetworking.googleapis.com \
--ranges=cloudsql-range --network=app-vpc
# 2. Cloud SQL instance: private IP only, no public IP, regional HA
gcloud sql instances create app-db \
--database-version=POSTGRES_16 --availability-type=REGIONAL \
--tier=db-custom-2-7680 --region=$REGION \
--network=projects/$PROJECT_ID/global/networks/app-vpc \
--allocated-ip-range-name=cloudsql-range \
--no-assign-ip --backup-start-time=03:00
gcloud sql databases create app --instance=app-db
DB_PASSWORD="$(openssl rand -base64 24)"
gcloud sql users create appuser --instance=app-db \
--password="$DB_PASSWORD"
# 3. Service account + least-privilege IAM (prefer secret-level binding)
gcloud iam service-accounts create run-sql-sa
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:run-sql-sa@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/cloudsql.client"
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:run-sql-sa@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/cloudsql.instanceUser"
printf '%s' "$DB_PASSWORD" | gcloud secrets create db-password \
--replication-policy=automatic --data-file=-
gcloud secrets add-iam-policy-binding db-password \
--member="serviceAccount:run-sql-sa@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
# 4. VPC connector that can route to the Cloud SQL private range
gcloud compute networks vpc-access connectors create sql-connector \
--network=app-vpc --region=$REGION --ip-cidr-range=10.8.0.0/28 \
--min-throughput=200 --max-throughput=1000
Deploy the service with the Auth Proxy as a sidecar from the manifest below:
gcloud run services replace service.yaml --region=$REGION
To use IAM database authentication instead of a password, set the cloudsql.iam_authentication=on database flag, add --auto-iam-authn to the proxy arguments, and register the service account as a database user (for PostgreSQL the user name is the email without .gserviceaccount.com). Then grant only the privileges the application needs:
GRANT CONNECT ON DATABASE app TO "run-sql-sa@example-project.iam";
The application then connects as run-sql-sa@example-project.iam with no password.
Production Failure Modes
Connection storms: Cloud Run scales out and each instance opens its own pool. Cap it with --max-instances and --concurrency, and use PgBouncer or a managed pooler in front of Cloud SQL. The Auth Proxy does not pool connections.
Connector throughput: a VPC connector saturates at its max throughput (1000 Mbps). Raise --max-throughput for large payloads.
IAM propagation delay: a new role binding can take up to a minute or two to take effect — a fresh deploy may briefly see 403 from the Admin API.
Timeouts: set driver connect timeouts and statement_timeout; Cloud Run request timeout tops out at 60 minutes, so long queries must be async.
Backups/DR: enable automated backups and PITR (shown in Terraform below). Cloud SQL does not fail over across regions automatically — use a cross-region replica for DR.
Monitoring: alert on cloudsql.googleapis.com/database/cpu/utilization, database/network/connections, database/disks/utilization, plus Cloud Run latency/error metrics; enable Cloud SQL Insights and VPC/Cloud SQL admin activity audit logs.
Best Practices
Private IP only (--no-assign-ip), TLS enforced, IAM-based auth, secrets in Secret Manager.
Grant roles/cloudsql.client and roles/secretmanager.secretAccessor — not roles/cloudsql.admin.
Set --max-instances and use a connection pooler; scale the Cloud SQL tier with the expected concurrency, not the average.
Pin the proxy image tag/digest and redeploy deliberately — the proxy is part of your data path.
For strict isolation, consider Private Service Connect for Cloud SQL (--enable-private-service-connect with --allocated-psc-range) instead of the peered range.
How do you design a scalable web application on Google Cloud Platform?
KnownRevise
A scalable GCP web application is typically a stateless tier (Cloud Run or GKE) behind a global HTTP(S) load balancer, with managed data stores (Cloud SQL, Spanner, Firestore), Memorystore for caching, Cloud Storage for static assets, and Pub/Sub for asynchronous work, all secured with least-privilege IAM and observed via Cloud Monitoring.
Read full answerHide full answer
Applies toGoogle Cloud Platform (current stable APIs and gcloud CLI)
Architecture Overview
Global external HTTP(S) load balancer terminates TLS, routes by host/path, and fronts Cloud CDN for cacheable content.
Stateless compute holds no local state; sessions live in Memorystore Redis or signed cookies.
Managed data layer handles durability, replication, and backups; the app tier scales horizontally.
Async processing via Pub/Sub decouples slow work (email, image processing, webhooks) from request latency.
Compute Choice
Service
Best for
Scaling model
Notes
Cloud Run
Stateless HTTP containers, event-driven
Request-based, scale to zero
Fastest to ship; cold starts possible
GKE (Autopilot)
Complex microservices, custom networking
HPA + cluster autoscaler
More control, higher ops cost
App Engine (standard)
Simple web apps, legacy PaaS
Automatic
Less flexible than Cloud Run
Compute Engine
Stateful/legacy, custom OS needs
MIG autoscaling
Most control, most ops burden
Choose Cloud Run for most new HTTP services; move to GKE when you need sidecars, custom ingress, or fine-grained networking.
Data Layer
Cloud SQL (PostgreSQL/MySQL): regional relational data with automated backups and read replicas.
Cloud Spanner: globally distributed, strongly consistent relational data at higher cost.
Firestore: document store for flexible schemas and real-time sync.
Bigtable: wide-column store for high-throughput time-series/IoT workloads.
Cloud Storage: static assets, uploads, backups; serve via Cloud CDN.
Networking and Load Balancing
Place compute in a VPC with private IPs; use Serverless VPC Access connectors so Cloud Run reaches Cloud SQL/Memorystore privately.
Attach Cloud Armor policies for WAF, rate limiting, and DDoS protection.
Use Cloud CDN in front of Cloud Storage and cacheable endpoints.
Prefer private service access for Cloud SQL instead of public IPs.
Scaling and Autoscaling
Cloud Run: set --concurrency, --min-instances (to avoid cold starts on critical paths), and --max-instances (cost guardrail); autoscaling is request/concurrency-based.
GKE: Horizontal Pod Autoscaler on CPU/memory/custom metrics plus cluster autoscaler; use Pod Disruption Budgets.
Database: scale reads via replicas; scale writes via sharding or Spanner; watch connection limits (use connection pooling, e.g., Cloud SQL Auth Proxy or PgBouncer).
Security
One service account per service with least-privilege IAM roles; avoid the default Compute service account.
Use Workload Identity on GKE and service account impersonation for CI/CD.
Store secrets in Secret Manager, mounted as env vars or files; never bake secrets into images.
Enable VPC Service Controls for sensitive data perimeters; enforce org policies (e.g., disable public IPs).
Sign and scan container images; deploy only from Artifact Registry.
Define SLOs and burn-rate alerts; use uptime checks on the load balancer.
Propagate trace context across services; log request IDs.
Cost Optimization
Scale to zero where latency allows; set max instances to cap spend.
Use committed use discounts for steady GKE/Compute baseline.
Cache aggressively at CDN and Redis; right-size Cloud SQL.
Avoid cross-region egress; keep compute and data in the same region.
Code Example
# Deploy a containerized service to Cloud Run with scaling and VPC access
gcloud run deploy web-api \
--image=us-central1-docker.pkg.dev/PROJECT_ID/repo/web-api:latest \
--region=us-central1 \
--allow-unauthenticated \
--concurrency=80 \
--min-instances=2 \
--max-instances=50 \
--cpu=1 --memory=512Mi \
--vpc-connector=projects/PROJECT_ID/locations/us-central1/connectors/run-connector \
--set-secrets=DB_PASSWORD=db-password:latest
Best Practices and Pitfalls
Do not store session state on instances; it breaks horizontal scaling.
Watch Cloud SQL connection limits; use pooling and the Cloud SQL Auth Proxy.
Test cold-start latency before relying on scale-to-zero for user-facing paths.
Keep regions consistent to avoid egress cost and latency; use multi-region only when justified.
Automate everything with Terraform; treat infra as code and review changes via CI.
76
How do you design a production data analytics pipeline on GCP?
KnownRevise
A production analytics pipeline on Google Cloud typically uses Pub/Sub for event ingestion, Dataflow for stream and batch processing, Cloud Storage for immutable raw data, and BigQuery as the analytical warehouse. This architecture supports both real-time dashboards and large-scale batch reporting while remaining independently scalable.
Read full answerHide full answer
Applies toGoogle Cloud CLI 500.0.0+ with current stable Google Cloud managed services
End-to-End Architecture
Sources publish events: Applications, APIs, databases, and operational systems publish to a versioned Pub/Sub topic. For example, an order service publishes order.created events containing an event_id, event timestamp, and business payload.
Pub/Sub buffers ingestion: Pub/Sub provides durable, at-least-once delivery and absorbs temporary traffic spikes. It decouples producers from consumers.
Dataflow validates and transforms: A Dataflow streaming pipeline consumes events, validates schemas, removes invalid records, enriches data, and writes curated results.
Cloud Storage preserves raw data: Dataflow or an ingestion service writes immutable raw objects to Cloud Storage. These objects provide a replay source for backfills and recovery.
BigQuery serves analytics: Curated events and dimensional tables are stored in partitioned BigQuery tables. Looker, Looker Studio, or internal APIs run dashboards, reports, and ad hoc analysis.
Batch processing uses the same model: Scheduled sources land in Cloud Storage. Cloud Scheduler and Workflows trigger a Dataflow batch job that produces additional BigQuery tables.
Service Selection
Capability
Google Cloud
AWS equivalent
Azure equivalent
Event ingestion
Pub/Sub
SNS/SQS or EventBridge
Event Grid or Service Bus
Stream processing
Dataflow
Kinesis Data Analytics, Glue, EMR
Stream Analytics or Fabric
Raw object storage
Cloud Storage
S3
Blob Storage
Analytical warehouse
BigQuery
Redshift or Athena
Synapse or Fabric
Semantic layer and BI
Looker
QuickSight
Power BI
Choose BigQuery for SQL analytics, especially when elasticity and large scans matter. Choose Bigtable for high-throughput, low-latency operational key-value access, not as a general-purpose SQL warehouse. Use Cloud Storage plus BigQuery external tables when retaining Parquet files is more important than materializing every dataset.
Data Model and Processing
Define an explicit data contract, preferably using a registered Avro or Protobuf schema. Reject incompatible changes before they reach downstream consumers.
Include an immutable event_id, an occurrence timestamp, and an ingestion timestamp. Use event_id for deduplication because Pub/Sub delivery is at least once.
Partition BigQuery tables by date and cluster by frequently filtered columns such as customer_id or event_type.
Keep raw, validated, and curated datasets separate. This prevents source alterations from silently changing analytical results.
Route poison messages to a Pub/Sub dead-letter topic. Retain enough metadata to identify the schema version, failure reason, and source.
Reliability, Security, and Cost
Reliability: Configure Dataflow checkpointing, bounded retries, and autoscaling. Acknowledge Pub/Sub messages only after successful downstream processing. Test recovery by replaying raw Cloud Storage data.
Availability: Deploy pipelines in the required region and, when justified, maintain an independently deployable failover job in another region. Do not assume automatic cross-region Dataflow failover.
Monitoring: Alert on Pub/Sub backlog age, undelivered message count, Dataflow job failures and stalled workers, BigQuery failures, and data-freshness SLOs. Also monitor row counts, null rates, duplicate rates, and late events.
Security: Use separate service accounts for ingestion and processing. Grant narrowly scoped roles such as roles/pubsub.publisher, roles/pubsub.subscriber, roles/storage.objectAdmin, roles/bigquery.dataEditor, and roles/dataflow.worker. Avoid primitive project-wide roles.
Data protection: Use CMEK where required, VPC Service Controls for sensitive workloads, private networking for Dataflow workers, and organization policies restricting public access.
Cost control: Partition and cluster BigQuery tables, limit repeated full-table scans, control Dataflow worker autoscaling, and apply Cloud Storage lifecycle rules. Use BigQuery slots when predictable consumption justifies committed capacity.
Code Example
The following creates the main managed resources and an initial partitioned BigQuery table. The project must already exist, have billing enabled, and the caller must have permission to enable APIs and create these resources.
Define freshness, completeness, latency, and availability SLOs before launch.
Test schema evolution, duplicate delivery, poison messages, regional failures, and partial Dataflow failures.
Version schemas and pipelines; deploy schema changes through backward-compatible migrations.
Use object versioning or retention policies for critical raw data, and regularly test restoration procedures.
Keep secrets in Secret Manager and avoid placing credentials in Pub/Sub messages, BigQuery rows, or Cloud Storage objects.
Maintain a replayable raw layer and documented backfill process. During an incident, preserve dead-letter messages rather than deleting them.
77
How do you secure a new Google Cloud organization?
KnownRevise
Securing a new GCP organization means locking down the resource hierarchy, IAM, and organization policies before any workload is deployed, then enabling centralized logging and guardrails that apply to every project created afterward.
Read full answerHide full answer
Applies toGoogle Cloud Platform (gcloud CLI 4xx, current stable)
Resource Hierarchy and IAM
Create the organization under a verified domain via Cloud Identity or Google Workspace; the organization is the root of all resources.
Use folders to separate environments (prod, staging, dev) and business units; apply IAM at the folder level so child projects inherit policy.
Grant roles/resourcemanager.organizationAdmin to a small break-glass group only; use custom roles or predefined roles scoped to folders/projects for everyone else.
Avoid primitive roles (Owner, Editor, Viewer) on the organization; prefer fine-grained predefined roles.
Enable Workload Identity Federation for CI/CD and external workloads instead of long-lived service account keys.
Organization Policies
Set constraints at the organization level so they inherit to all projects: disable service account key creation (constraints/iam.disableServiceAccountKeyCreation), require OS Login (constraints/compute.requireOsLogin), restrict VM external IPs, and require Shielded VMs.
Restrict resource locations with constraints/gcp.resourceLocations to keep data in approved regions.
Disable creation of default networks and require VPC flow logs via policy where supported.
Use Policy Intelligence (IAM Recommender) to remove unused permissions over time.
Identity and Access
Enforce MFA and SSO through Cloud Identity/Workspace; disable consumer Gmail accounts for org resources.
Use groups, not individual users, in IAM bindings; rotate membership via your IdP.
Create a dedicated break-glass project and account with Organization Administrator, monitored and alerted on use.
Logging, Monitoring, and Detection
Admin Activity audit logs are always on; explicitly enable Data Access audit logs for sensitive services (Cloud Storage, BigQuery, Secret Manager).
Create an organization-level log sink to a central, tamper-resistant project or BigQuery dataset for long-term retention and SIEM ingestion.
Enable Security Command Center (Premium where budget allows) for misconfiguration and threat detection.
Set up alerting on IAM policy changes, new service account keys, and firewall rule changes.
Network and Data Protection
Use VPC Service Controls perimeters around sensitive projects to prevent data exfiltration.
Default-deny firewall rules; allow only required ports and sources.
Encrypt data at rest with CMEK via Cloud KMS for regulated workloads; manage keys in a separate project.
Use Secret Manager for credentials; never store secrets in source code or environment variables in plaintext.
Code Example
# List organizations
gcloud organizations list
# Disable service account key creation org-wide
gcloud resource-manager org-policies enable-enforce \
constraints/iam.disableServiceAccountKeyCreation \
--organization=ORG_ID
# Require OS Login on Compute instances
gcloud resource-manager org-policies enable-enforce \
constraints/compute.requireOsLogin \
--organization=ORG_ID
# Restrict resource locations
gcloud resource-manager org-policies set-policy policy.yaml --organization=ORG_ID
# Organization-level audit log sink to a central GCS bucket
gcloud logging sinks create org-audit-sink \
storage.googleapis.com/central-audit-bucket \
--organization=ORG_ID \
--include-children \
--log-filter='logName:"cloudaudit.googleapis.com"'
Apply guardrails at the organization/folder level, not per project, so new projects inherit them automatically.
Do not grant roles/owner to developers; use project-level roles/editor only where justified and time-boxed.
Test organization policies in a non-prod folder first; some constraints break existing automation.
Grant the log sink’s service account the required writer role on the destination bucket or dataset.
Review SCC findings and IAM Recommender suggestions on a recurring schedule, not just at setup.
Keep a documented break-glass procedure and test it quarterly.
AWS/Azure Equivalents
GCP
AWS
Azure
Organization + Folders
AWS Organizations + OUs
Management Groups
Organization Policies
SCPs
Azure Policy
Cloud Identity / Workspace
IAM Identity Center
Entra ID
VPC Service Controls
(no direct equivalent)
Private Link + NSGs
Security Command Center
Security Hub / GuardDuty
Microsoft Defender for Cloud
Choose GCP organization policies when you want declarative, inheritable constraints; combine with folders for environment isolation and SCC for continuous posture monitoring.
78
How do I troubleshoot 503 errors on a Google Cloud Run service?
KnownRevise
A Cloud Run 503 service-unavailable error usually means either no healthy container can serve the request or the application itself returned 503. Troubleshoot by separating platform availability failures from application, dependency, and rollout failures.
Read full answerHide full answer
Applies toGoogle Cloud Run managed services (current stable) and gcloud CLI 2.x
Triage Workflow
Confirm the URL and status code
A wrong URL normally returns 404.
Authentication or IAM failures normally return 403, not 503.
Test both the service root and an application-owned health endpoint.
Inspect service and revision state
Check the active image, environment configuration, service account, resource limits, traffic assignments, and status conditions.
Compare the failing revision with the last known-good revision.
Inspect the service IAM policy if callers receive 403 responses.
Read logs
Inspect container logs for crashes, missing environment variables, failed dependency calls, port-binding errors, startup-probe failures, and explicit 503 responses.
Check Cloud Logging for errors from Cloud Run revisions.
Check production metrics
In Cloud Monitoring, review request count by status code, request latency, container instance count, CPU, and memory.
A sudden drop in ready instances, repeated restarts, or memory saturation explains many platform-generated 503s.
What 503 Means Under the Hood
Cloud Run routes incoming requests through the Google Front End to an available revision instance. A container must listen on the port supplied through the PORT environment variable and bind to 0.0.0.0, not only a loopback address. If a startup probe is configured, it must also succeed before the instance receives traffic.
A platform-generated 503 commonly indicates:
Every instance has crashed, failed its startup checks, or otherwise never became ready.
The container is listening on the wrong address or port.
Memory exhaustion is causing out-of-memory restarts.
A revision has no ready instance.
The service configuration points to an image that cannot start.
The application can also deliberately return 503 when a database, cache, external API, or other required dependency is unavailable. Application-generated 503s normally have corresponding application logs. Exceeding the CPU limit normally causes throttling and increased latency rather than an automatic restart.
Cloud Run’s default request timeout is 300 seconds. Exceeding the request timeout normally produces a 504, although an intermediary or application may translate dependency timeouts into 503.
Correct the startup dependency and create a new revision
Startup probe never succeeds
Incorrect probe path, dependency still unavailable, or unhealthy startup
Fix the probe or startup dependency
Error mentions binding or PORT
Incorrect listen address or port
Listen on 0.0.0.0:$PORT
Process exits immediately
Bad entrypoint, command, image, or incompatible architecture
Verify the container command and image locally
Application logs show a 503 response
Dependency or business-health failure
Check database, VPC, DNS, quota, and dependency latency
Errors affect only the newest revision
Bad deployment or configuration change
Shift traffic back to the known-good revision
Instances restart with out-of-memory errors
Memory pressure or leak
Profile memory, adjust limits, and fix the leak
Code Example
#!/usr/bin/env bash
set -euo pipefail
PROJECT="example-project"
REGION="us-central1"
SERVICE="example-service"
HEALTH_PATH="/healthz"
# Show configuration, conditions, traffic, and the active image.
gcloud run services describe "$SERVICE" \
--region="$REGION" \
--project="$PROJECT" \
--format=yaml
# Inspect the service IAM policy.
gcloud run services get-iam-policy "$SERVICE" \
--region="$REGION" \
--project="$PROJECT"
# List revisions and their traffic state.
gcloud run revisions list \
--service="$SERVICE" \
--region="$REGION" \
--project="$PROJECT"
# Read recent logs and recent errors for this service's revisions.
LOG_FILTER="resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"${SERVICE}\""
gcloud logging read "$LOG_FILTER" \
--project="$PROJECT" \
--freshness=1h \
--limit=200
gcloud logging read "${LOG_FILTER} AND severity>=ERROR" \
--project="$PROJECT" \
--freshness=1h \
--limit=200
URL="$(gcloud run services describe "$SERVICE" \
--region="$REGION" \
--project="$PROJECT" \
--format='value(status.url)')"
# The identity token works for a private service and is also accepted by a public one.
TOKEN="$(gcloud auth print-identity-token)"
for request_path in "/" "$HEALTH_PATH"; do
curl --max-time 15 -i \
-H "Authorization: Bearer ${TOKEN}" \
"${URL}${request_path}"
done
If the latest revision is broken, identify an exact healthy revision from gcloud run revisions list and restore traffic to it:
PROJECT="example-project"
REGION="us-central1"
SERVICE="example-service"
REVISION="example-service-00041-abc" # Replace with an exact revision ID from the list.
gcloud run services update-traffic "$SERVICE" \
--to-revisions "${REVISION}=100" \
--region="$REGION" \
--project="$PROJECT"
Traffic shifting is an immediate mitigation, not a repair. Diagnose the failed revision before redeploying.
Production Readiness
Configure Cloud Monitoring alerts for 5xx rates, latency, instance count, CPU, and memory.
Use Cloud Trace and structured application logs to identify slow dependencies.
Keep the runtime service account least-privileged; startup failures may result from missing IAM bindings.
Grant access to Secret Manager resources only when required, and never print secret values in incident logs.
Validate private connectivity and Cloud SQL or VPC connectivity from the service’s environment.
Treat successful deployment as insufficient: readiness proves that the container startup contract succeeded, including any configured startup probe, not that business endpoints are healthy.
Increasing minimum instances can reduce cold-start latency, but it does not fix crashing containers and increases cost.
79
How do you set up CI/CD for a containerized application on GCP?
KnownRevise
A typical GCP CI/CD flow for containers uses Cloud Build to build and test images, Artifact Registry to store them, and Cloud Deploy to promote releases across environments to Cloud Run or GKE. Keeping builds, artifacts, and deployments inside GCP lets you enforce IAM and audit everything in one place.
A push to your source repository (GitHub, GitLab, or Bitbucket; Cloud Source Repositories is closed to new customers) triggers a Cloud Build trigger.
Cloud Build runs tests, builds the container image, and pushes it to Artifact Registry.
Cloud Deploy creates a release and rolls it out through configured targets (dev, staging, prod), with optional manual approvals between stages.
The target runtime (Cloud Run or GKE) pulls the new image and runs the container.
Key Components
Cloud Build: managed build service driven by a cloudbuild.yaml file.
Artifact Registry: regional repository for Docker images (replaces Container Registry).
Cloud Deploy: managed continuous delivery with promotion, approvals, and rollback.
Runtime: Cloud Run for serverless containers, GKE for Kubernetes workloads.
Pipeline Stages
Trigger on push to main.
Run unit and integration tests.
Build image tagged with the commit SHA.
Push image to Artifact Registry.
Create a Cloud Deploy release.
Auto-deploy to dev; require approval to promote to prod.
Runtime Comparison
Factor
Cloud Run
GKE
Ops overhead
None (serverless)
Cluster management
Scaling
Automatic, to zero
HPA / cluster autoscaler
Best for
Stateless HTTP services
Complex, stateful, or multi-service workloads
Networking
Managed ingress
Full VPC control
Security and Best Practices
Use Workload Identity Federation for GitHub/GitLab; avoid long-lived service account keys.
Enable Artifact Analysis (container scanning) and block vulnerable images.
Apply least-privilege IAM: separate build, deploy, and runtime service accounts.
Use distroless or minimal base images and pin versions.
Enable Binary Authorization to enforce attestation before deploy.
Tag images with commit SHA and never overwrite latest in production.
Code Example
Create the repository and submit a build:
gcloud artifacts repositories create my-repo \
--repository-format=docker --location=us-central1
# COMMIT_SHA is populated automatically only for trigger-based builds;
# a manual submit leaves it empty, so run this file through a trigger.
gcloud builds submit --config=cloudbuild.yaml