TLDR: A URL shortener is a read-heavy system (100:1 ratio) that maps long URLs to short, unique aliases. The core scaling challenge is generating unique IDs without database contention—solved using a Range-Based ID Generator or a Distributed Counter with Base62 encoding. A 301/302 redirect strategy combined with aggressive Redis caching ensures sub-10ms response times.

💥 The "Duplicate Link" Disaster

Imagine you're building a URL shortener for a massive marketing campaign. Two different users, one in Paris and one in Tokyo, both click "Shorten" at the exact same microsecond.

If your system uses a naive random string generator (e.g., randomString(6)), there's a small but real chance both users get the same code: bit.ly/AbC123.

1. User A maps it to shoes.com/sale.
2. User B maps it to malware-site.net/steal-data.

If User B's write happens second and overwrites User A, everyone clicking the "Shoe Sale" link is now being redirected to a malware site. This is a Short-Code Collision. In a system handling 10,000 writes per second, "one-in-a-billion" chances happen every few days. You cannot rely on luck; you need a deterministic way to guarantee uniqueness across a distributed cluster of servers.

📖 URL Shortening: Use Cases & Requirements

Actors

• Link Creator: Submits a long URL and receives a compact short link.
• Link Clicker (End User): Clicks the short link and is redirected to the destination.
• Marketer / Admin: Views analytics (click counts, geo-data) and manages link expiry.

Functional Requirements

• URL Shortening: Generate a unique, short alias for any given long URL.
• Redirection: Redirect users from the short link to the original long URL with minimal latency.
• Custom Aliases: Allow users to specify their own short strings (e.g., bit.ly/my-awesome-link).
• Link Expiry: Support optional expiration dates for short links.
• Analytics: Track click counts, referrers, and basic browser/geo metadata.

Non-Functional Requirements

• High Availability: Redirections must never fail (99.999% availability).
• Low Latency: Redirects should happen in < 10ms for a cache hit.
• Scalability: Support 100M+ new links per month and billions of redirects.
• Uniqueness: Short codes must never collide.

🔍 Basics: Base62 and the Counter Strategy

The simplest way to ensure uniqueness is to treat every short code as a number. If we have a unique counter that increments for every new link, we just need to convert that number into a "URL-friendly" string.

We use Base62 Encoding ([0-9, a-z, A-Z]).

• A 6-character string in Base62 can represent $62^6 \approx 56.8$ Billion unique URLs.
• A 7-character string can represent $62^7 \approx 3.5$ Trillion unique URLs.

The process is: Counter (Long) -> Base62 Encode -> Short String. For example, the number 200,000,000 becomes 1LpUvA. This is deterministic, collision-free, and highly efficient.

⚙️ Mechanics: Distributed ID Generation (The Ticket Server)

To scale the counter across multiple servers, we cannot have every server increment a single row in a SQL database—that would be a massive bottleneck. Instead, we use a Range-Based Allocation (similar to Flickr's "Ticket Server"):

1. A central coordinator (Zookeeper or a dedicated DB table) manages "Ranges" (e.g., 1-1000, 1001-2000).
2. When a Shortener Service instance starts, it requests a range (e.g., "Give me 1,000 IDs").
3. The server handles the next 1,000 requests entirely in memory.
4. Once it hits the end of its range, it requests a new one.

If a server crashes, the remaining IDs in its range are "lost" (unused), which is perfectly fine given we have 56 billion available.

📐 Estimations & Design Goals

Capacity Planning

• Write Volume: 100M links / month $\approx 40$ links per second.
• Read Volume: 10B redirects / month $\approx 4,000$ redirects per second.
• Storage: 100M links 500 bytes per record = 50GB per month. Over 5 years, we need *3TB of storage.
• Cache: Following the 80/20 rule (20% of links generate 80% of traffic), we should cache the top 20% of active links.

Design Goal: Optimize for Read Throughput. Since redirects outnumber creations by 100:1, we must ensure that the "Read Path" never hits the database under normal conditions.

📊 High-Level Design: The Redirect-First Architecture

The following architecture prioritizes redirection speed while ensuring background analytics don't slow down the user.

graph TD
    User((User)) --> LB[Load Balancer]
    LB --> AG[API Gateway]

    subgraph Write_Path
        AG --> CS[Creation Service]
        CS --> KGS[ID Generator: Zookeeper]
        CS --> PDB[(Primary DB: Postgres)]
    end

    subgraph Read_Path
        AG --> RS[Redirect Service]
        RS --> RC[(Redirect Cache: Redis)]
        RS -.->|Fallback| PDB
    end

    subgraph Analytics_Pipeline
        RS --> Kafka[Kafka]
        Kafka --> AS[Analytics Service]
        AS --> DW[(Data Warehouse: ClickHouse)]
    end

The diagram shows two primary traffic flows. On the Write Path, a request travels from the Load Balancer through the API Gateway to the Creation Service, which claims an ID range from ZooKeeper before persisting to Postgres. On the Read Path — which carries 99% of all traffic — the Redirect Service checks Redis first and only falls back to Postgres on a cache miss. The Analytics Pipeline is fully decoupled from both paths: click events stream through Kafka to the Analytics Service asynchronously, so analytics processing never adds latency to a user's redirect experience.

🧠 Deep Dive: How Range Allocation Eliminates the Counter Bottleneck at Scale

Internals: How the Range Allocation Coordinator Manages ID Namespaces

ZooKeeper (or etcd) stores a single persistent counter node at /shortener/id_ranges/next_range. Each Shortener Service instance, on startup, executes a single atomic compareAndSet(currentValue, currentValue + RANGE_SIZE) operation. If the CAS succeeds, the instance owns the range [currentValue, currentValue + RANGE_SIZE) exclusively and increments a local in-memory counter for every subsequent ID request. If the CAS fails (another instance claimed the range first), the instance retries with the freshly read value. This optimistic locking approach is both fast (single round-trip per range claim) and correct (no two instances ever receive the same range).

ZooKeeper persists its state to a majority-quorum WAL, meaning the range counter survives ZooKeeper node failures and application restarts. If a Shortener Service instance crashes mid-range, the abandoned IDs in that range are simply skipped — the global namespace has 56 billion entries and can absorb billions of abandoned IDs without issue.

Performance Analysis: Latency Budget for the 100:1 Read/Write Ratio

The system must serve redirects at sub-10ms p95 latency. Breaking down the latency budget for the read path:

At a 95%+ cache hit rate, the p95 redirect path budget is under 4ms. The 5% cache miss path (hitting Postgres) runs at 10–20ms — still within SLA but significantly slower. The design goal is to keep the cache hit rate above 95% by setting appropriate TTLs and proactively warming cache for links crossing a traffic threshold.

Redis memory sizing: With 10 million active links in cache and an average 200-byte Redis entry (key + value + metadata), the redirect cache requires approximately 2GB of Redis RAM — easily handled by a 3-node Redis Cluster with 8GB per node.

The naive approach to unique ID generation — a single auto-incrementing counter in Postgres — breaks down immediately in a multi-server environment. Every ID request requires a read-then-write on the counter row, protected by a pessimistic lock. At 40 writes per second across 20 application servers, you serialize 800 lock acquisitions per second through a single database row. This is a textbook hot-key bottleneck.

Range Allocation solves this by inverting the coordination pattern. Instead of polling for the next ID on every request, each server claims a batch of IDs upfront using a single atomic update: UPDATE id_ranges SET next_range = next_range + 1000 WHERE current_range = :claimed RETURNING next_range. The server then increments a local in-memory counter for the next 1,000 requests without any network communication.

When a server crashes mid-range, the remaining IDs in that range are abandoned — a non-issue given the 56 billion total capacity of 6-character Base62 codes.

Base62 Namespace Capacity by String Length

Six characters is universally sufficient. A 7-character namespace will not be required in any realistic production scenario within our lifetimes.

Data Model: The Links Table

The custom_alias column shares a unique index with short_code. Before accepting a custom alias, the creation service must verify uniqueness. Unlike range-allocated IDs, custom aliases carry collision risk and must be rejected with a 409 Conflict if already taken.

Redis Cache Schema: The Read-Path Engine

The click:buffer key is critical for viral links. Without buffering, a link receiving 50,000 clicks per second would trigger 50,000 Postgres UPDATE click_count = click_count + 1 operations per second — an impossible write rate for a single row. Redis's atomic INCR batches these into one bulk update every 5 minutes.

🌍 Real-World Applications: Bitly, Twitter t.co, and Pinterest

Bitly processes over 10 billion clicks per month across more than 300 million unique shortened links. Their production architecture uses ZooKeeper for range coordination across the shortener fleet. Bitly's analytics pipeline runs on Apache Flink for real-time stream processing, consuming Kafka click events and materializing hourly and daily aggregations into Apache Druid. Bitly offers both 301 and 302 redirects as a user-level configuration option, allowing marketers to trade click-tracking accuracy for server cost savings.

Twitter's t.co wraps every external URL tweeted through the platform in a t.co short link. Unlike Bitly, t.co is invisible to users — it operates at the infrastructure level to enable URL safety scanning and click attribution within Twitter's ad platform. t.co uses a hash-based approach rather than a counter, as link creation happens asynchronously during tweet processing. All t.co links use 301 redirects to push redirect traffic to browser caches and dramatically reduce server load.

Pinterest's URL shortener is an internal tool used for sharing pins via SMS and push notifications. Pinterest's analysis found that the top 1% of shortened links account for over 60% of all redirect traffic. As a result, they implemented a "hot link" promotion strategy: links crossing a traffic threshold are proactively pre-warmed into Redis across all cache nodes, achieving a 97%+ cache hit rate during viral pin events.

⚖️ Trade-offs and Failure Modes Every Interviewer Expects You to Discuss

301 vs 302: A Joint Product and Infrastructure Decision

Custom Alias Race Condition

Two concurrent requests for the same alias (e.g., bit.ly/launch-day) will both read "not exists" before either completes its write, potentially causing a collision. The mitigation is a UNIQUE constraint on the custom_alias column: the database enforces uniqueness at the write level, and the losing request receives a 409 Conflict and retries with a different alias.

Cache Staleness on Link Deactivation

When a link is deactivated (malware flagged, DMCA takedown), the database is_active flag is set to false. However, the Redis cache still holds url:{short_code} with the original long_url for up to 24 hours. Production systems must pair every database deactivation with an explicit DEL url:{short_code} Redis command to force immediate cache eviction. Without this, deactivated links continue redirecting through the full TTL window — a serious security gap.

🧭 Decision Guide: Choosing the Right ID Strategy for Your Scale

🧪 Interview Delivery Example: Narrating a URL Shortener End-to-End in 45 Minutes

A URL shortener interview tests three specific conversations: ID uniqueness at scale, the 100:1 read/write ratio response, and analytics isolation. Here is how a senior engineer structures the session.

Minute 1–5: Requirements scoping. Ask: "Are custom aliases required? Is analytics real-time or batch acceptable? Is there a link retention policy?" This signals design maturity and avoids overbuilding.

Minute 6–15: ID generation deep dive. Avoid "random string" — jump to the counter approach immediately. Say: "I would use a range-based counter with Base62 encoding. Each server claims a block of 1,000 IDs from a ZooKeeper-managed counter at startup, then increments an in-memory counter. This gives deterministic uniqueness without any per-request locking, and Base62 with 6 characters covers 56 billion unique links — enough for 47 years at 100 million new links per month."

Minute 16–30: Architecture walkthrough. Present the two-path architecture: "The write path runs through the Creation Service to ZooKeeper for ID allocation, then Postgres for persistence. The read path hits Redis first — the 100:1 read/write ratio means 99% of redirect traffic never touches Postgres. On a cache miss, we fall back to Postgres and repopulate Redis."

Minute 31–40: Analytics and failure modes. Say: "Click tracking is fully asynchronous. The Redirect Service emits a click event to Kafka without waiting for a delivery acknowledgment. An Analytics Consumer processes these into ClickHouse for aggregation. This decouples analytics latency from redirect latency entirely — the user gets their redirect in under 10ms regardless of how long analytics processing takes." Then proactively address failure: "If Redis goes down, a circuit breaker opens and all redirects fall back to Postgres at higher latency. We alert immediately and restore Redis, then warm the cache from Postgres before closing the circuit."

Minute 41–45: Trade-off discussion. Expect questions on 301 vs 302, custom alias collision handling, and link deactivation cache staleness. Have specific, concrete answers for each — interviewers are checking for operational depth, not just architecture diagrams.

🛠️ Redis, Kafka, and ZooKeeper: How the Production Stack Fits Together

Redis (Caching and Click Buffering): Redis Cluster with 6 nodes (3 primary, 3 replica) distributes the redirect cache across 16,384 hash slots. The short code is hashed to determine which node handles each lookup. Redis's O(1) GET/SET and configurable TTL make it the standard redirect cache substrate. The atomic INCR command enables click buffering without requiring distributed transactions.

Apache Kafka (Analytics Pipeline): Kafka decouples click events from analytics processing. The Redirect Service publishes lightweight ClickEvent messages (short_code, timestamp, referrer, user_agent, ip) to a url.clicks topic with 30-day retention. An analytics consumer group reads from this topic and batch-writes to ClickHouse, which handles sub-second aggregate queries across billions of click rows via columnar compression. Kafka's retention means analytics consumers can replay events after recovery from failures without data loss.

Apache ZooKeeper (Range Coordination): ZooKeeper's ephemeral nodes track which ID ranges have been claimed. On startup, each Shortener Service instance creates an ephemeral node under /shortener/ranges and atomically increments the next_range counter, claiming the resulting block. On shutdown or crash, the ephemeral node disappears. A ZooKeeper watch triggers re-allocation if a node is idle too long. This approach survives restarts and partial cluster failures without ID duplication.

📚 Lessons Learned from Production URL Shortener Systems

The 99/1 Rule, Not 80/20. In production deployments, the top 1% of links receive over 95% of redirect traffic. Standard LRU caching handles this well under steady state, but viral link events — a celebrity post, a product launch — can cause sudden cache misses on previously cold links. Production teams implement traffic spike detection that proactively pre-warms cache entries for links crossing a sudden request-rate threshold, preventing the cold-start penalty during viral moments.

Expiry Causes Thundering Herds. Bulk-expiring millions of links at scheduled times (midnight, campaign end date) creates a simultaneous Postgres DELETE operation storm combined with Redis cache invalidation spikes. Use soft-delete with is_active = false and run a background janitor with rate-limited deletes during off-peak hours to spread the load.

Custom Aliases Create Ongoing Support Burden. Users create custom aliases, forget they own them, and open support tickets claiming the alias "was never set up." Always build alias ownership transfer workflows with email verification, or plan to handle constant alias-dispute tickets for the lifetime of the platform.

301 vs 302 Has Direct Infrastructure Cost Implications. Switching from 302 to 301 for non-analytics links reduced redirect server capacity requirements by 25–30% at multiple companies because browsers cache the redirect client-side. The product and infrastructure teams must make this decision jointly — it is not purely a technical choice.

📌 Key Takeaways: URL Shortener System Design

• A URL shortener is a read-heavy system (100:1 read/write ratio). Optimize every design decision for the redirect read path first; write performance is secondary.
• Base62 encoding with 6 characters covers 56 billion unique URLs — sufficient for any real-world deployment over the next 47 years.
• Range-Based Allocation solves the distributed counter bottleneck: each server claims a block of IDs from ZooKeeper and increments an in-memory counter locally, with zero per-request network coordination.
• Redis caching with a 24-hour TTL and LRU eviction is the standard read-path optimization. Target a 95%+ cache hit rate in steady state.
• Analytics must be asynchronous. Publish click events to Kafka and process in the background — never allow analytics latency to affect the user's redirect response time.
• Link deactivation requires active cache invalidation. Always pair a database is_active = false update with a Redis DEL url:{short_code} to prevent stale cache serving deactivated links.
• 301 vs 302 is a joint product and infrastructure decision that simultaneously affects analytics accuracy, browser caching, server cost, and SEO.

🔗 Related Posts

• System Design HLD: Distributed Rate Limiter
• System Design HLD: Notification Service Architecture
• System Design HLD: Web Crawler Architecture