Presigned URLs (Direct-to-Storage Uploads)
Cloud infrastructurecloud infrastructure / storage
Symptom 1: the API server is the bottleneck
Your flow: browser → POST /upload (your API) → s3.put_object() → S3.
Fine for 2 MB avatars. Then users start uploading 500 MB videos:
- pods OOMKilled — the framework buffers the body in RAM
- p99 on unrelated endpoints jumps — 20 slow uploads pin every worker
- transfer billed twice: in to your compute, out to S3
Reason it through
The bytes never needed to touch your server. Its only real job was authorization: "may user 42 write videos/42/clip.mp4?" You're spending RAM, threads and bandwidth to babysit a byte pipe.
The fix: sign, don't proxy
A presigned URL is an ordinary S3 URL with a signature of your server's credentials in the query string. Whoever holds it can perform exactly one operation (PUT that key), for a limited time, without ever having your keys.
url = s3.generate_presigned_url(
"put_object",
Params={"Bucket": "uploads",
"Key": f"user/{uid}/{uuid4()}.mp4",
"ContentType": "video/mp4"},
ExpiresIn=900) # 15 minutes
// browser
await fetch(url, {method: "PUT",
headers: {"Content-Type": "video/mp4"},
body: file})
Server RAM per upload: ~0. Worker threads held: 0.
Symptom 2: "works in curl, 403 in the browser"
SignatureDoesNotMatch. Causes, most common first:
1️⃣ Header mismatch. You signed Content-Type: video/mp4, the browser sent video/mp4; charset=utf-8. Every header in Params becomes part of the signature — send it back byte-identical, or don't sign it at all.
2️⃣ CORS. The PUT is cross-origin, so the bucket must allow PUT from your origin (and expose ETag for multipart). A missing rule often surfaces as a vague network error, not a clean 403.
3️⃣ Expiry / clock skew. The signer's clock drifting 10 min makes URLs dead on arrival.
4️⃣ The URL got mangled — re-encoded, truncated in a log, or passed through a redirect.
What practitioners actually get wrong
The URL is the authorization. Once issued you cannot revoke it, and nothing inspects the file afterwards.
- Sign a key YOU generate. Never presign a client-supplied path, or I'll PUT to user/1/avatar.png.
- A presigned PUT cannot cap file size. If that matters, use a POST policy with content-length-range.
- Treat the object as untrusted until a post-upload step (S3 event → Lambda) checks real type/size and flips your DB row to ready. "Upload finished" ≠ "file is safe".
- It's a bearer credential: don't log it, don't cache it in a CDN, keep ExpiresIn in minutes.
- Same trick works for downloads — presigned GET beats streaming private files through your API.
Your exercise
Presign a PUT with ExpiresIn=60 and upload with:
curl -X PUT --upload-file ./test.mp4 "<url>"
Then (a) wait 90s and retry the same URL, and (b) inside the window, retry with Content-Type changed by one character.
Two different failures. Before you run them — can you predict which returns ExpiredToken vs SignatureDoesNotMatch, and which error body names the header that broke it?