Design the upload of profile pictures and large videos. How do the bytes reach storage?

The API authenticates the user and returns a presigned URL, and the client uploads straight to object storage. Large files use multipart upload with chunks and checksums so they can be resumed. A storage event then triggers processing such as thumbnails. Events arrive at least once, so workers must be idempotent. The trade-off is more steps for the client, in exchange for servers that never carry the bytes.

Where do you keep the file name, owner and permissions, and why not in the object store?

In a database, because it supports queries, joins and transactions, and object storage does not. The object store holds the bytes under a key. The row links to the key. The trade-off is keeping two systems consistent, which needs an order of writes and a periodic clean-up of orphans. Write the object first and then the row, so a crash leaves a harmless orphan object rather than a row pointing at nothing.

How do you make sure an uploaded file is never lost?

Use a store that keeps several copies across zones, turn on versioning so an overwrite or delete can be undone, and for critical data replicate across regions. Verify checksums on upload. The trade-off is storage cost for the extra copies and versions.

Users should only download their own files. How do you enforce that?

Keep buckets private. After checking permission in the API, hand out a short-lived signed URL for that single object (one method, one key, a short expiry), or serve through a CDN that validates signed requests. The trade-off is that a link works for anyone who has it until it expires, so keep lifetimes short.

A user edits a small part of a 2 GB file. Why is object storage a poor fit, and what do you do?

Objects are immutable, so changing a byte means writing a whole new object. For frequent small edits, store the file in chunks and rewrite only the changed chunk, or use a file system or block storage. The trade-off is complexity in tracking chunks against the simplicity of whole-object writes.

Blob and Object Storage | TheLearningLoop

Why you need object storage, and when to use it

Photos, videos, documents, backups and logs are large files that are written once and read many times. Storing them in a database makes it huge and slow, and storing them on one server's disk risks losing them and limits how many can be kept. Object storage keeps each file as an object (the bytes plus metadata) under a key in a bucket, spread across many machines and copied for durability.

It gives you two things interviewers always ask about:

Durability: objects are copied across machines or zones, so a failed disk does not lose data.
Scalability and cost: capacity grows almost without limit at a low price per gigabyte, and the application servers stay stateless.

Use it when:

You store user uploads, media, backups, logs or datasets.
Files are large and read as a whole, or by ranges.
You want to serve files through a CDN straight from storage.
Data must survive hardware failures cheaply.

Skip it, or use something else, when:

You need fast random updates inside a file or transactions across files: use a database or a file system.
Objects are tiny and queried by content: use a database.
You need very low latency for hot data: put a cache in front or use a block volume.

Try it

Upload a 100 MB file over a network that can fail. Compare one request against chunks.

Blob and Object Storage

Step 1 of 4 · Press Space to play, arrow keys to step

UploadFailure

Initial state. A 100 MB file must reach the object store over a network that can fail.

tick:: 0
MB sent:: 0
wasted MB:: 0
attempts:: 1

1 tick = one step of the simulation.

Speed

Terms used in the counters: wasted MB is data sent that had to be sent again, attempts counts upload tries, and a tick is one step of the simulation. A CDN is a network of caches near users, a queue holds messages for background workers, and a lifecycle rule is an automatic rule such as "delete abandoned parts after 7 days".

Things to try:

Run One request with Connection drops at 70%. The whole upload restarts and 70 MB are wasted.
Switch to Chunks with the same failure. Only the lost chunk is sent again.
Choose One piece arrives damaged with both. See how a checksum per chunk limits the repair.

Storage choices

Kind	How it stores data	Good when	Weakness
Object storage	Immutable objects by key, over HTTP	Media, backups, data lakes	No in-place edits, higher latency than a disk
Block storage	Raw disk volumes attached to a server	Databases and operating systems	Tied to one machine, you manage the file system
File storage	A shared file system with folders and locks	Shared files for several servers	Harder to scale, more expensive
Database blob column	The file inside a row	Tiny files that need a transaction	Bloats the database and backups

Direct upload vs uploading through your servers

Through your servers: the client sends the file to your API, which forwards it to storage. It is simple, but your servers carry all the bytes and become the bottleneck.
Direct upload with a presigned URL: your API checks permission, then gives the client a time-limited signed URL. The client uploads straight to storage, and your servers handle only small requests. This is the usual choice at scale.

How a large upload works

The client asks your API for permission to upload and receives a presigned URL (or a set of them) valid for a few minutes.
For large files the client starts a multipart upload and splits the file into chunks of a few megabytes.
Each chunk is uploaded on its own, possibly in parallel, with a checksum. The store rejects a chunk whose checksum does not match.
If the connection drops, the client asks which chunks are stored and resumes with the missing ones.
When all chunks are in, the client completes the upload and the store joins them into one object.
The store sends an event (for example to a queue) so other services can process the file, such as making thumbnails. Events are usually delivered at least once, so a worker may see the same event twice and must be idempotent (doing it twice gives the same result).
Abandoned partial uploads are cleaned up by a lifecycle rule.

Object storage vs the alternatives

Concern	Object storage	Database blobs	Server disk
Capacity	Practically unlimited	Limited	Limited
Durability	Very high, copied	High	Low
Cost per gigabyte	Low	High	Medium
Edit part of a file	No	Yes	Yes
Serve through a CDN	Easy	Awkward	Awkward
Best for	Files and media	Tiny files	Temporary data

Object storage wins for files. Keep the metadata (owner, name, size, permissions) in a database and only the bytes in object storage.

What a presigned URL allows

A presigned URL is a bearer credential: anyone who holds it can use it until it expires. Scope it tightly.

One method (PUT or GET), one key, and a short expiry.
Where the provider supports it, restrict size and content type (for example with signed headers or a POST policy), and still validate the file after upload.
Give each user a key prefix you control, never a name the client chooses.

A rough estimate

A 100 MB file split into 10 MB chunks is 10 requests, plus one to start and one to complete. If the connection drops at 70%, a single request resends 100 MB (170 MB sent in total), while chunks resend at most the chunk in flight. Larger chunks mean fewer requests but more resent data on a failure; smaller chunks mean the opposite. Providers set limits on chunk size, number of chunks and object size (Amazon S3, for example, allows up to 10,000 parts per object), so check your provider's documentation.

Avoiding a new single point of failure

Durability comes from the provider copying objects across zones. For critical data, add cross-region replication and versioning, which keeps older copies when an object is overwritten or deleted.
Access: serve through a CDN so a storage hiccup does not break every page.
Metadata database: replicate it. An object with no metadata row is effectively lost, so write the row and the object in a safe order and reconcile regularly.

You accept higher latency than a local disk and no in-place edits in exchange for durability and scale. Consistency rules differ between providers (Amazon S3 has been strongly consistent for reads and lists since 2020, but older or self-hosted systems may not be), so check yours.

When the object and the row disagree

Step order	Crash between the two writes	Repair
Write the object, then the row	An orphan object with no row (wasted money, no user impact)	A periodic job lists objects and deletes those with no row after a grace period
Write the row, then the object	A dangling row pointing at nothing (users see a broken file)	Harder: mark the row `pending` and only show it once the object exists

Prefer object first, then row, and reconcile regularly.

Common mistakes

Streaming uploads through application servers. Use presigned URLs.
One huge request for big files. Use multipart upload with resume.
Public buckets by accident. Default to private and serve through signed URLs or a CDN.
Trusting the file name or type. Validate content, size and type, and scan for malware.
No lifecycle rules. Abandoned uploads and old versions pile up and cost money.
No link between the row and the object. Deleting one and not the other leaves orphans.

How to talk about object storage in an interview

Say the files go to object storage, with metadata in a database and a CDN for reads.
Describe the presigned URL flow and multipart upload with checksums and resume.
Mention durability (copies, versioning, replication), access control, lifecycle rules and events for processing.
Name the trade-off: objects are cheap and durable but cannot be edited in place, so a change means writing a new object.

Where you meet object storage in real systems

Almost every product that accepts uploads uses it for profile photos, attachments, exports and backups. Here are three drawn out, the first with a short script you can run, then more examples.

Example 1: Multipart upload and a lifecycle rule

Object stores accept a large file as numbered chunks, check each chunk and join them when asked. The script below shows the idea with a dictionary standing in for the store. Open the example below to see the picture and the code.

Checksummed, resumable chunksSetup a short scriptCode no libraries

The API checks permission and hands out a signed URL. The client uploads chunks straight to storage, and the store joins them. An event then triggers processing.

# Split a file into chunks, give each a checksum, and resume after a failure.
# The "store" below is a dictionary that stands in for an object store. No libraries are needed.
import hashlib

CHUNK = 4                                   # bytes per chunk, tiny so the output is easy to read
data = b"hello object storage"
store = {}                                  # chunk number -> (bytes, checksum the server accepted)

def upload_chunk(number, chunk, checksum, damage=False):
    sent = chunk[:-1] + b"?" if damage else chunk          # simulate corruption on the wire
    if hashlib.sha256(sent).hexdigest() != checksum:        # the server verifies every chunk
        return False                                        # rejected: only this chunk is sent again
    store[number] = (sent, checksum)
    return True

chunks = [data[i:i + CHUNK] for i in range(0, len(data), CHUNK)]
sent_count = 0
for number, chunk in enumerate(chunks):
    checksum = hashlib.sha256(chunk).hexdigest()            # computed by the client before sending
    ok = upload_chunk(number, chunk, checksum, damage=(number == 2))   # chunk 2 arrives damaged once
    sent_count += 1
    if not ok:
        ok = upload_chunk(number, chunk, checksum)          # resend just that chunk
        sent_count += 1
    assert ok

# Resume: after a dropped connection, ask which chunks are stored and send only the missing ones.
missing = [n for n in range(len(chunks)) if n not in store]

joined = b"".join(store[n][0] for n in sorted(store))        # "complete the upload": join the chunks
assert joined == data and not missing
print(len(chunks), "chunks,", sent_count, "sends")

The Python script was run with Python 3: one damaged chunk was rejected and resent, and the joined result matched the original. It shows the idea only: it damages one chunk and resends it, but it does not simulate a dropped connection, so the resume step just lists the missing chunks. Real S3-style stores offer the same steps as multipart upload calls, which I could not test here, so check your provider's documentation for the exact calls.

Example 2: Metadata and bytes live apart

The database holds the information about the file. Object storage holds the bytes. A queue connects them to background work.

The client uploads bytes directly to object storage. The API records metadata in a database. A storage event goes on a queue so a worker can make thumbnails.

Example 3: Chunks of a large upload

Each chunk has its own checksum. A damaged or lost chunk is sent again without touching the others.

Chunks 1 to 7 are stored. Chunk 8 was interrupted. The client resumes at chunk 8, and chunks 1 to 7 are never sent again.

More real-world scenarios

4. Profile pictures

The app asks for an upload URL, the client uploads directly, and a worker creates several sizes. The page uses a CDN URL.

5. Document and attachment storage

Files belong to users or teams. The database keeps the owner and permissions, and downloads use short-lived signed links.

6. Backups and archives

Databases and logs are copied to cheap storage, with rules that move old data to colder, cheaper tiers.

7. Data lakes

Raw events are written as files, and analytics engines read them straight from storage when they run queries.

8. Static websites and assets

Pages, images and scripts are served from a bucket through a CDN, with no servers to run.

9. Video and audio

Originals and transcoded segments are objects, and the CDN serves them to players.

10. Software releases

Build outputs are stored once with a version and fetched by deployment tools.

Tools and managed services

Kind	Name	Notes
Managed	Amazon S3, Google Cloud Storage, Azure Blob Storage	Hosted object storage
Self-hosted	MinIO, Ceph	S3-compatible storage you run yourself
CDN	Amazon CloudFront, Cloudflare	Serve objects near users
Tools	AWS CLI, MinIO client (mc)	Upload, share and manage objects
Events	Storage event notifications to queues	Trigger processing when an object arrives

The takeaway

Whenever a design stores files, put the bytes in object storage, the metadata in a database, and let clients upload straight to storage with a signed URL.

Blob and Object Storage

Key takeaways

Why you need object storage, and when to use it

Try it

Blob and Object Storage

Storage choices

Direct upload vs uploading through your servers

How a large upload works

Object storage vs the alternatives

What a presigned URL allows

A rough estimate

Avoiding a new single point of failure

When the object and the row disagree

Common mistakes

How to talk about object storage in an interview

Where you meet object storage in real systems

Example 1: Multipart upload and a lifecycle rule

Example 2: Metadata and bytes live apart

Example 3: Chunks of a large upload

More real-world scenarios

4. Profile pictures

5. Document and attachment storage

6. Backups and archives

7. Data lakes

8. Static websites and assets

9. Video and audio

10. Software releases

Tools and managed services

The takeaway

Interview questions