The day a "images aren't showing" bug dragged me from the frontend down to Google Cloud org policies

I had built a feature that generates product images with AI. One day I noticed the gallery was empty — not even a "generating" indicator. I started thinking "five-minute bug," and ended up wiring together Google Cloud org policies, IAM, Vertex AI, and Cloudflare AI Gateway. Here's the whole thing, start to finish, with every wall I hit along the way.

The start: "No images in the gallery"

My app (sayfa.app) has a Studio feature: the user picks a product photo, tweaks the scene/lighting/style, and AI generates a professional product image. Behind the scenes it runs Google's Gemini image model (gemini-3-pro-image).

My complaint was simple:

"AI-generated images don't show up in the gallery. And when I hit 'generate', there's no 'generating' indicator anywhere."

The first diagnosis pointed at the frontend. The gallery never refreshed itself when a generation completed. The code was invalidating the cache when the generation started (while the image didn't exist yet) — and with the wrong query key on top of that. So even if the image was generated in the background, it wouldn't appear in the gallery until you manually reloaded the page.

The fix was clean: I added an effect that invalidates the gallery + media queries when a generation reaches completed (for both single and multi-variation). I also fixed the "generating" indicator and the error message.

I thought it was done here. It wasn't.

The first real wall: 429 "prepayment credits are depleted"

To see if the fix actually worked, I ran a generation live. I picked a product image, hit "Generate." "Generating…" showed up — good, the indicator worked. Then:

{
  "error": {
    "code": 429,
    "message": "Your prepayment credits are depleted. Please go to AI Studio ... to manage your project and billing.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

This is where the real bomb went off. The actual reason images weren't showing in the gallery wasn't the frontend bug — no generation could ever complete, because Google was rejecting every single request with a 429.

So I started ruling things out, the boring but necessary way:

Was the API key correct? Yes. I confirmed the key value, that the same value was sitting in the Kubernetes secret, and that the running pod was using that exact key. There was nothing wrong with the key.
Was the pod using the new secret? The pod had started after the secret update, the live env value was correct, it didn't even need a restart.

Everything on my side was correct. The 429 wasn't a config mistake — it was coming from how the AI Studio Gemini API (generativelanguage.googleapis.com) was set up on Google's side. And on the forums there were plenty of people hitting the exact same 429 with this API. The key authenticated fine (listing models worked); only the actual generation call failed.

The direction of the fix: Vertex AI

Research pointed at a clear door. The same Gemini models are reachable through two different paths:

AI Studio / Gemini API (generativelanguage.googleapis.com) — the simple API-key path. This is the one that was failing.
Vertex AI (aiplatform.googleapis.com) — the enterprise path, tied to the Google Cloud project itself.

The naming was confusing (Google even renamed Vertex to "Gemini Enterprise Agent Platform"), but the takeaway was simple: both routes serve the same Gemini models, but they're wired into the project differently — and the Vertex path was the one I needed.

But before writing any blind code, I tested. With a temporary token I fired a generation request straight at Vertex:

[us-central1] → 404 NOT_FOUND
[global]      → 200 OK, image generated ✓

Second surprise here: the gemini-3-pro-image model is in no regional endpoint at all, only on the global endpoint. If I'd written the code to us-central1 without knowing this, I'd have wrestled with 404 for hours. A real lemon photo (and a blue coffee mug too) got generated — no errors. Vertex being the fix was proven.

The second wall: authentication

Vertex doesn't accept API keys — it wants OAuth / service account. I had to decide how the Kubernetes pod would authenticate to Vertex in production. There were three paths:

Service account JSON key — simple, but...
Workload Identity Federation (WIF) — no JSON key needed, most secure
Cloudflare AI Gateway — Cloudflare handles the Google auth + cache/log/analytics bonus

The third wall: an org policy bans JSON keys

I tried to create a JSON key:

Enforced Organization Policy: iam.disableServiceAccountKeyCreation
"Key creation is not allowed on this service account."

My organization had banned service account key creation for security (actually a good default). First I wondered "is this managed from admin.google.com?" — no, this is a Google Cloud policy, managed from console.cloud.google.com.

Then I realized: I didn't have permission to change this policy. Which meant Cloudflare's BYOK (Bring Your Own Key) method — since it requires a JSON key — was also locked.

The fourth wall (and the turning point): WIF is hard on self-hosted k3s too

I turned to WIF — it didn't require JSON. But when I looked at my cluster's OIDC issuer:

issuer: https://kubernetes.default.svc.cluster.local

That's a cluster-internal address, unreachable from the internet. For WIF to work, Google needs to reach that metadata. There was a way around it (uploading a static JWKS, without touching the API server at all) but the setup was a fair bit of work.

The actual fix: I gave myself the permission

The machine I was working on had no gcloud installed — so I did all of this against Google's REST APIs directly, with curl and an OAuth access token. Here are the exact calls, in order.

First, the honest question: what can I actually do here? testIamPermissions answers it without trial-and-error.

# Which of these permissions do I hold on the project?
curl -s -X POST \
  "https://cloudresourcemanager.googleapis.com/v3/projects/${PROJECT_ID}:testIamPermissions" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d '{"permissions":["orgpolicy.policy.set","iam.serviceAccountKeys.create"]}'
# → returned only what I had. orgpolicy.policy.set was NOT in the list.

So I couldn't touch the key-creation policy at the project level. But checking the organization itself revealed I had org-level setIamPolicy:

curl -s -X POST \
  "https://cloudresourcemanager.googleapis.com/v3/organizations/${ORG_ID}:testIamPermissions" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d '{"permissions":["resourcemanager.organizations.setIamPolicy"]}'
# → permission present. I'm effectively the org owner.

Being able to set the org IAM policy meant I could grant myself the missing role.

Step 1 — grant myself roles/orgpolicy.policyAdmin. Read the current org policy, add a binding, write it back:

# get current org IAM policy
curl -s -X POST \
  "https://cloudresourcemanager.googleapis.com/v3/organizations/${ORG_ID}:getIamPolicy" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' -d '{}' \
  > org-iam.json

# (locally) append a binding: roles/orgpolicy.policyAdmin → user:me@example.com
# then push it back
curl -s -X POST \
  "https://cloudresourcemanager.googleapis.com/v3/organizations/${ORG_ID}:setIamPolicy" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d @org-iam-with-new-binding.json
# → 200, binding applied.

A few seconds later, testIamPermissions confirmed orgpolicy.policy.set was now mine.

Step 2 — disable the key-creation ban, but only on THIS project (the org-wide rule stays enforced everywhere else). This is a single Org Policy API call that creates a project-level override with enforce: false:

curl -s -X POST \
  "https://orgpolicy.googleapis.com/v2/projects/${PROJECT_ID}/policies" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d "{
    \"name\":\"projects/${PROJECT_ID}/policies/iam.disableServiceAccountKeyCreation\",
    \"spec\":{\"rules\":[{\"enforce\":false}]}
  }"
# → 200. Then I verified the effective policy actually flipped:
curl -s \
  "https://orgpolicy.googleapis.com/v2/projects/${PROJECT_ID}/policies/iam.disableServiceAccountKeyCreation:getEffectivePolicy" \
  -H "Authorization: Bearer ${TOKEN}"
# → "enforce": false  ✓

Step 3 — create the service account and give it the Vertex role:

# create the service account
curl -s -X POST \
  "https://iam.googleapis.com/v1/projects/${PROJECT_ID}/serviceAccounts" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d '{"accountId":"sayfa-studio-vertex","serviceAccount":{"displayName":"Sayfa Studio Vertex AI"}}'

# grant it roles/aiplatform.user (read project IAM, add binding, set it back — same pattern as step 1)
curl -s -X POST \
  "https://cloudresourcemanager.googleapis.com/v1/projects/${PROJECT_ID}:setIamPolicy" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d @project-iam-with-aiplatform-user.json

Step 4 — create the JSON key. This is the exact call that used to fail with "Key creation is not allowed on this service account" — now it works:

curl -s -X POST \
  "https://iam.googleapis.com/v1/projects/${PROJECT_ID}/serviceAccounts/sayfa-studio-vertex@${PROJECT_ID}.iam.gserviceaccount.com/keys" \
  -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  -d '{"privateKeyType":"TYPE_GOOGLE_CREDENTIALS_FILE"}'
# → the JSON key comes back base64-encoded in "privateKeyData"; decode it to get the file.

One catch worth flagging: org-policy changes take a couple of minutes to propagate. The first key-creation attempt right after disabling the policy still returned "not allowed." I polled it, and ~60 seconds later it succeeded. Then I tested the key end-to-end exactly as production would (service account → mint an access token via the JWT-bearer flow → call Vertex). A photo of a green plant in a white pot came back. ✓

The final piece: Cloudflare AI Gateway

I now had a JSON key, so I could take the most useful path: Cloudflare AI Gateway. The reason wasn't just auth convenience — over the course of this day I'd seen I desperately needed exactly these things:

Analytics → to see how many requests/tokens each call spends (I'd spent all day blind to what was actually happening)
Logging → to see which request failed and why (I'd chased the 429 through pod logs)
Cache → so the same translation doesn't go to Google over and over → savings
Rate limit + retry → against abuse and sudden cost spikes

I created an AI Gateway in Cloudflare, added the service account JSON as BYOK (so no Google credential ever enters the pod — just a single static Cloudflare token). The Cloudflare docs said "we don't recommend the global region," but since my model only exists on global, I tried it anyway:

Cloudflare AI Gateway → Vertex → global → 200 OK, image generated ✓

It worked. Both the image model and the text model came back fine over global.

Bonus win: I made translation 22x cheaper

While inspecting the code on this journey, I noticed that for simple text jobs like template translation, an expensive model (gemini-3.5-flash, ~$9 per 1M tokens) was being used. For translation that's pure waste. I switched to the cheapest model (gemini-2.5-flash-lite, ~$0.40) — about 22x cheaper, and the quality is more than enough for translation.

The result

The final architecture:

Pod  →  (only a static CF token)  →  Cloudflare AI Gateway  →  (BYOK auth)  →  Vertex AI (global)  →  image

The pod carries no Google credential.
Generation works through the Vertex path — no more 429.
Cache, log, analytics, rate limit come for free from Cloudflare.
Translation is 22x cheaper.

What I learned along the way

The same model can be reached two ways, and they behave differently. AI Studio's Gemini API and Vertex AI serve the same models but are wired into your project differently. When one path keeps throwing 429 with a valid key, try the other path before assuming your config is broken.
Measure before you write code. I tested the question "will Vertex fix this?" in 5 minutes with a temporary token. If it hadn't, I'd have wasted hours writing code for nothing.
gemini-3-pro-image only exists on the global endpoint. 404 on regional endpoints. Testing this upfront saved me a ton of time.
Org policies aren't there for no reason. The JSON key ban felt like an obstacle, but it's actually a sound security default. WIF is the ideal; but in practice I loosened the policy only on a single project and balanced it with Cloudflare's model that keeps the credential away from the pod.
The real bottleneck is rarely where you think. What started as a "frontend gallery bug" turned into a billing/auth/infrastructure chain. The symptom and the root cause are rarely in the same place.

I thought it was "a small frontend fix" one afternoon. In the end I wired together org policy, IAM, Vertex AI, and Cloudflare AI Gateway — and when I saw a lemon photo get generated, I don't think I'd ever been so happy.

productdevbook/from-a-gallery-bug-to-google-cloud-org-policies.md

Select an option

No results found