ShrinkRay All Articles
Developer Productivity

Your API Is Oversharing: How to Stop Sending Data Nobody Asked For

By ShrinkRay Developer Productivity
Your API Is Oversharing: How to Stop Sending Data Nobody Asked For

Picture a waiter who, when you order a coffee, brings you the coffee, a full breakfast spread, the lunch menu, three dessert options, and a printed copy of the restaurant's entire ingredient sourcing policy. Technically, you got what you asked for. Also, your table is now unusable.

This is what most REST APIs do on every single request.

The GET /users/{id} endpoint returns the full user object — all 47 fields of it — because that's what was convenient to build. The mobile client that called it needed the user's display name and avatar URL. The other 45 fields hit the network, got deserialized, sat in memory for a moment, and were immediately discarded. That's not a hypothetical. That's happening in your production environment right now, thousands of times per minute.

The Root Cause: Convenience-Driven API Design

Over-fetching doesn't happen because developers are careless. It happens because returning entire objects is genuinely easier to build and maintain. You write one endpoint, it returns the whole model, and every client can pull whatever it needs. No coordination required.

The problem is that this convenience is paid for in bandwidth, latency, and memory — and those costs are paid by your users and your infrastructure bill, not your development velocity metrics.

The pattern compounds over time. An API that launched with a lean user model grows as the product grows. New fields get added to the database table, they get added to the ORM model, and because the endpoint just serializes the model, they automatically show up in every API response. Nobody audits this. Nobody asks whether the iOS app needs the last_password_change_timestamp field it's been receiving for two years.

Measuring What You're Actually Sending

Before optimizing anything, you need to understand the scope of the problem. A few approaches:

Log response payload sizes in production. Add middleware that records the byte size of every API response alongside the endpoint path. Aggregate this over a week and sort by total bytes transferred (size × request count). The results are often genuinely shocking. A single high-traffic endpoint returning bloated objects can account for 30–40% of your total API egress.

Sample responses and count fields. Write a script that hits each of your major endpoints and counts the fields in the response. Then cross-reference with your frontend code to see which fields are actually accessed. Tools like jq make this easy for quick audits. For a more systematic approach, API gateways like Kong or AWS API Gateway can log request/response pairs that you can analyze offline.

Measure with real client behavior. In your frontend code (or mobile app), add instrumentation that tracks which response fields are actually read before the response object is garbage collected. This is the ground truth. Everything else is inference.

One team that went through this exercise with a social platform API found that their /feed endpoint — their highest-traffic endpoint by a significant margin — was returning 23 fields per post object. Client-side instrumentation showed that the feed UI consumed exactly 6 of them. The other 17 were making the trip for no reason.

After stripping unused fields, their average feed response dropped from 84KB to 31KB. That's a 63% reduction in bandwidth on their most-called endpoint.

REST Approaches: Field Selection Without Blowing Up Your API

You don't have to abandon REST to fix over-fetching. Several patterns work well without requiring a full architectural overhaul:

Sparse fieldsets (the JSON:API approach). Allow clients to request specific fields via query parameters: GET /users/123?fields=name,avatar_url,email. The server parses the fields parameter and only serializes the requested attributes. This is explicit, cacheable, and requires no client-side changes beyond adding the parameter.

Implementation note: Be careful about caching. A response to GET /users/123?fields=name and GET /users/123?fields=name,email are different resources. Make sure your cache keys include the field selection, or you'll end up serving stale partial responses.

Multiple endpoint variants. For high-traffic endpoints where you know the use cases, create explicit lean variants: GET /users/{id}/summary returns the minimal representation, while GET /users/{id} returns the full object. More endpoints to maintain, but crystal-clear contracts and excellent caching behavior.

Response shaping middleware. Some API gateways and frameworks support response transformation rules that strip specified fields before responses leave the server. This is useful for quick wins on existing APIs without changing backend code — but it's a band-aid, not a solution. The serialization cost is still paid; you're just trimming the output.

GraphQL: The Promise and the Pitfall

GraphQL was explicitly designed to solve over-fetching. Clients declare exactly what fields they need, and the server returns exactly that. In theory, it's perfect.

In practice, GraphQL implementations frequently over-fetch at the resolver level even when they're not over-delivering to the client. If your User resolver fetches the full user row from the database and then GraphQL trims it to the requested fields, you've still done the database work for every field. The wire transfer is lean; the backend is not.

For GraphQL to genuinely eliminate over-fetching, your resolvers need to be field-aware — only querying the database columns that correspond to the requested fields. This is non-trivial to implement correctly but the performance payoff is real. Libraries like join-monster (for SQL backends) can help automate this.

Also watch for the N+1 problem in GraphQL — requesting a list of items and then resolving nested fields triggers individual database queries per item. DataLoader is the standard solution here, but it requires deliberate implementation.

A Practical Audit Checklist

If you're ready to start trimming, here's a reasonable sequence:

  1. Identify your top 10 endpoints by total bytes transferred. This is where the wins are.
  2. For each endpoint, list every field in the response. Then list every field actually consumed by each client type (web, iOS, Android, internal services).
  3. Calculate the waste ratio. Fields sent ÷ fields used. Anything above 3:1 is a priority target.
  4. Implement sparse fieldsets or create lean endpoint variants for the top 3 offenders. Measure the before/after.
  5. Add response size logging to your monitoring dashboard. Make the waste visible so it doesn't creep back.
  6. Update your API design guidelines to require field necessity justification for any new endpoint. New fields should be opt-in, not opt-out.

The Compounding Return

Over-fetching isn't just a bandwidth problem. Smaller responses parse faster, which means faster time-to-interactive on the client. They're cheaper to log and store. They reduce memory pressure on both server and client. They make your API contracts clearer and easier to reason about.

Every unnecessary field you stop sending is a small win. Multiplied across millions of requests, those small wins show up in your infrastructure costs, your app's performance scores, and your users' experience — especially on mobile connections where bandwidth is genuinely constrained.

Your API knows a lot. That doesn't mean it needs to say all of it.