HLD Roadmap

A. Philosophy

HLD is not "memorize the architecture of Twitter." HLD is the skill of turning a vague product requirement into a system that meets its scale, reliability and cost constraints, and of explaining why each component exists and what breaks when load or requirements change.

Total: about 70 hours over 7 weeks (10 hours per week), roughly 100 to 110 substantial lessons. Prerequisite: LLD fundamentals (objects, interfaces, dependency direction) and basic programming. Learners who finished the ProgVale LLD path are ready.

B. Phase 1: Foundations and system thinking (weeks 1 to 4)

Week 1: Thinking in systems

Goal: Move from "I know how to build an app" to "I know how to reason about a system under load." Time: about 10 hours, 9 lessons.

Module 1: Requirements, scale and estimation

Small exercises

• Estimate QPS and storage for Twitter-like posting
• Estimate bandwidth for a video platform
• Convert "99.9% availability" into allowed downtime
• Estimate cache size for a read-heavy product catalog
• Estimate servers needed for 10K requests per second

For each exercise, identify: → assumptions → read/write ratio → peak vs average load → storage growth → the bottleneck

Critical exercise Give the learner: "Design a system that serves a profile page to 10 million users." Do not draw boxes first. Ask:

• What is the read/write ratio?
• What is the peak QPS?
• What is the data size per user?
• What fails first when traffic doubles?
• What is the cheapest thing that could possibly work?

This builds the habit of asking numbers-first questions before touching architecture.

Week 2: Networking, APIs and data foundations

Goal: Understand how clients reach systems and how data is stored and accessed. Time: about 10 hours, 9 lessons.

Module 2: Networking and API design

Worked example Requirement:

Design the API for a ride-booking app. Riders request rides. Drivers accept them. A ride has one driver at a time.

The learner should arrive at something like:

POST /rides                 (rider creates request, idempotency key)
POST /rides/{id}/accept     (driver accepts)
GET  /rides/{id}            (status polling or push)

Then ask:

• What happens if the client retries POST /rides after a timeout?
• Who enforces "one driver at a time"?
• Polling, SSE or WebSocket for status updates, and why?
• What changes if mobile apps cannot update immediately?

This moves the learner from endpoint listing to contract and failure reasoning.

Exercises

• URL Shortener API
• Payment API with idempotency
• Chat API (REST + WebSocket)
• Paginated feed API

Week 3: Data storage

Goal: Learn to choose, model and scale data stores. Time: about 10 hours, 9 lessons.

Module 3: Databases and storage

Teaching format (applies to every storage topic)

Problem
 ↓
Single database
 ↓
Where it hurts (size, load, failure)
 ↓
Technique (replicate / shard / index / denormalize)
 ↓
New problems introduced
 ↓
Improved design
 ↓
When NOT to apply it

Do not teach "use NoSQL for scale." The learner should feel the pressure that forces each choice.

Worked example Bad design: one MySQL instance holds users, orders, sessions, logs and search. Ask what differs per workload:

• Sessions: tiny, hot, expire quickly
• Orders: relational, transactional
• Logs: append-only, huge
• Search: full-text, ranked

Now polyglot persistence becomes intuitive, along with the cost of operating many stores.

Week 4: Scaling building blocks

Goal: Learn the standard components used to scale and protect systems. Time: about 10 hours, 12 lessons.

Module 4: Building blocks

Phase 1 integration project: Design a URL Shortener At the end of week 4, do not jump into complex systems. Give one open-ended design exercise.

Requirements:

• Create short links, redirect with low latency
• Custom aliases and expiry
• Basic click analytics
• 100M new links per month, 100:1 read/write

The learner must produce:

• Requirements and assumptions
• Estimations
• API
• Data model
• High-level architecture diagram
• Scaling plan (cache, sharding, ID generation)
• Failure discussion
• Trade-offs and what they would change at 10x scale

This combines the first four weeks.

Phase 1 outcome: a strong hold on fundamentals. The learner can take a small product requirement, estimate it, and produce a reasonable architecture.

C. Phase 2: Distributed systems and real HLD (weeks 5 to 7)

Complex systems begin here. The sequencing is deliberate: a learner should understand why distributed systems are hard before assembling one.

Week 5: Distributed systems and architecture patterns

Goal: Understand correctness, failure and the patterns that manage them. Time: about 10 hours.

Do not teach every distributed-systems theorem equally.

Concept tiers

This prevents spending equal time on every academic topic.

Learning order: start with failure, not theory.

Module 5: Distributed systems fundamentals

Module 6: Architecture styles

Component lesson structure Every component article contains exactly this structure:

1. Problem
2. Naive solution
3. Why the naive solution breaks
4. Forces / trade-offs
5. Component or technique
6. How it works internally
7. Placement in the architecture
8. Real-world example
9. When to use
10. When NOT to use
11. Failure modes
12. Recognition clues
13. Similar alternatives
14. Small exercise
15. Recall questions

Week 6: Component combinations and complete HLD

Goal: Stop thinking in isolated components. Real systems combine them. Time: about 10 hours.

Module 7: Component combinations

Every combination lesson follows:

Why one component isn't enough
 ↓
Responsibilities
 ↓
Data flow
 ↓
Failure paths
 ↓
Trade-offs

Complete HLD problem progression Do not make every problem a 45-minute interview exercise. Each problem follows:

Understand
 ↓
Estimate
 ↓
Design v1
 ↓
Find bottlenecks
 ↓
Scale
 ↓
Handle failure
 ↓
Explain

Level 1: Beginner

Level 2: Intermediate

Level 3: Advanced HLD

Level 4: Design engineer These are less about standard interview templates and more about genuine architectural ability.

Week 7: Reliability, operations and advanced trade-offs

Goal: Learn to evaluate, operate and evolve designs rather than simply create them. Time: about 10 hours, plus mock interviews.

Failure analysis and operations are an essential part of the curriculum, not an optional appendix.

Module 8: Design smells and anti-patterns

Each lesson follows:

Bad design
 ↓
Identify smell
 ↓
Why it happens
 ↓
Failure it causes
 ↓
Fix
 ↓
Improved design

Module 9: Reliability, operations and advanced thinking These lessons stay at the architecture level and do not turn into a DevOps course.

Phase 2 outcome: a strong hold on cross-component design. The learner can meet an unfamiliar product requirement and estimate, design, critique, scale and evolve a system.

D. Diagramming curriculum

Diagrams are taught alongside design, not as a separate week of tool training. They are limited to the practical subset needed for HLD.

How to choose: "What are the components?" leads to a block diagram. "How does data move?" leads to a data flow diagram. "What happens over time or on failure?" leads to a sequence diagram. "Where does it run?" leads to a deployment diagram.

E. The requirement-to-architecture algorithm

This is the central mental model of ProgVale HLD. Every serious design exercise forces the learner through it, and it should appear repeatedly throughout the course.

1. Requirements
       ↓
2. Scope and use cases
       ↓
3. Non-functional requirements
       ↓
4. Estimation
       ↓
5. API
       ↓
6. Data model
       ↓
7. Access patterns
       ↓
8. High-level architecture
       ↓
9. Storage choices
       ↓
10. Scaling the bottleneck
       ↓
11. Caching and async
       ↓
12. Consistency decisions
       ↓
13. Failure handling
       ↓
14. Observability
       ↓
15. Trade-offs
       ↓
16. Evolution

The learner eventually should not ask, "Which architecture does this question require?" They should ask:

• What are the scale and access patterns?
• What is the bottleneck?
• What must be consistent and what can be eventual?
• What happens when this component fails?
• Where is the single point of failure?
• Do I actually need this component yet?

That is the real HLD skill.

F. Small design exercise bank

These exercises sit between theory and complete HLD problems. Each reinforces one specific concept rather than being an arbitrary interview question.

G. The ProgVale lesson format

Every article follows one consistent template that fits the 10 to 25 minute format and the Learn, Practice, Recall, Apply, Integrate model.

Title: {Lesson Title}

Learning Objective
  By the end of this lesson you should be able to...

Prerequisites
  You should already understand...

The Problem
  Start with a real system problem with numbers.

Concept
  Explain the theory.

Example
  Show a small architecture or flow.

Bad Design
  Show the naive architecture.

Why It Is Bad
  Explain the failure under load, failure or change.

Better Design
  Evolve the architecture.

Design Thinking
  Explain how we arrived at the design.

Small Exercise
  Estimate / sketch...

Can You Explain?
  1. ...
  2. ...
  3. ...

Can You Design?
  Given...

Completion Criteria
  You are done when you can...

Recall
  Review this concept later.

H. Spaced learning architecture

This is where ProgVale can be better than a static HLD course. Concepts must not be learned once and then never seen again.

Avoid:

Caching → Sharding → Queues → CAP → never seen again

Instead, resurface each concept across later work:

Learn Caching
   ↓
Small cache exercise
   ↓
URL Shortener
   ↓
News Feed
   ↓
E-commerce
   ↓
Video Streaming
   ↓
Failure-analysis exercise (cache stampede)

The same applies to trade-offs:

Consistency vs Availability
   ↓
Key-Value Store
   ↓
Chat System
   ↓
Payment System
   ↓
Ticket Booking
   ↓
Global Low-latency Service

I. Dependency maps

Concept dependency graph

                    Programming + LLD Basics
                              │
                              ▼
                 Client-Server and Request Lifecycle
                              │
              ┌───────────────┼────────────────┐
              ▼               ▼                ▼
          Networking        APIs         Estimation
              │               │                │
              └───────────────┼────────────────┘
                              ▼
                  Single-server Architecture
                              │
              ┌───────────────┼────────────────┐
              ▼               ▼                ▼
          Databases        Caching        Load Balancing
              │               │                │
              ▼               │                │
   Replication / Sharding     │                │
              │               │                │
              └───────────────┼────────────────┘
                              ▼
                 Async Processing (Queues, Streams)
                              │
                              ▼
                  Distributed Systems Fundamentals
                  (failure, consistency, consensus)
                              │
                    ┌─────────┴─────────┐
                    ▼                   ▼
          Reliability Patterns   Architecture Styles
                    │                   │
                    └─────────┬─────────┘
                              ▼
                Component Combinations
                              │
                              ▼
                  Complete HLD Problems
                              │
                              ▼
               Failure Analysis and Anti-patterns
                              │
                              ▼
        Operations, Evolution and Advanced Trade-offs
                              │
                              ▼
                        Interview Ready
                              │
                              ▼
                     Strong HLD Engineer

Component dependency graph

Components have their own prerequisites. This is more educational than a flat list of "system design topics."

Estimation → Load balancing → Stateless services → Horizontal scaling
Single DB → Indexing → Replication → Sharding → Consistent hashing
Replication → Consistency models → Quorum → Leader election
Caching → Invalidation → Stampede protection → Multi-layer caching
Queues → Delivery semantics → Idempotency → Saga → Outbox
Queues → Logs/Kafka → Stream processing → Real-time analytics
Blob storage → CDN → Transcoding pipeline → Video streaming
Geohash/Quadtree → Proximity search → Ride-hailing
Inverted index → Search service → CDC sync → Search engine

J. Priorities: what to know

Must know deeply

Should know

Leader election and quorum, event sourcing, CQRS, outbox pattern, bloom filters, geospatial indexing, search and inverted indexes, stream vs batch processing, multi-region design, observability, SLOs, disaster recovery.

Awareness

Paxos/Raft internals, CRDTs, vector clocks, gossip protocols, two-phase commit internals, columnar and OLAP engine internals. The learner should recognize these and understand their purpose, but should not spend disproportionate time memorizing implementations.

K. Time allocation and phase summary

70-hour allocation

The allocation is intentionally weighted toward reasoning, failure and application, rather than spending 25 hours memorizing the architecture of famous products.

Phase 1 vs Phase 2

L. Interview progression

The learner moves through four stages. ProgVale should optimize for Stage 4.

M. What typical HLD courses get wrong

1. They teach famous systems before fundamentals

Typical course order:

Design Twitter → Design YouTube → Design Uber → ...

The learner memorizes diagrams but cannot adapt when the requirement changes. ProgVale uses this order instead:

numbers → bottleneck → failure → technique → component → system

2. They skip estimation

"Just add a cache and shard the database" means nothing without numbers. A design with no QPS, storage or bandwidth estimate is decoration. Estimation is taught in week 1 and used in every problem.

3. They teach CAP as a slogan

"Pick two of three" teaches nothing. Instead:

Two regions
      ↓
Network partition
      ↓
Accept writes on both sides?
      ↓
What does the user see?
      ↓
What is the cost of being wrong?
      ↓
Consistency model chosen per feature

4. They list technologies instead of trade-offs

"Use Kafka" or "use Cassandra" is not an answer. A learner must repeatedly answer: why this, what does it cost, and what is the simpler alternative?

Where should the feed be computed?

• On write (fan-out on write)?
• On read (fan-out on read)?
• Hybrid for celebrities?

There is not always one universal answer. The learner must learn to reason about the trade-offs.

5. They do not teach when NOT to scale

Bad HLD education teaches:

Load balancer → Microservices → Kafka → Kubernetes → Multi-region (for 1,000 users)

Good HLD teaches: do not add a component until a number or a failure mode demands it. This connects directly to YAGNI and avoiding overengineering at the system level.

6. They ignore failure

Most courses show the happy path. Real systems encounter:

slow dependency → retries → overload → cascading failure → outage

Failure analysis therefore gets its own module, not a closing remark.

7. They do not teach design evolution

The most important question is not "does this design work?" but "what happens at 10x load or when a requirement changes?" Every major ProgVale problem therefore has:

Version 1 (works for 10K users)
   ↓
Traffic grows 100x / new requirement
   ↓
What breaks first?
   ↓
Scale or redesign
   ↓
Version 2

This is probably the single strongest differentiator to build into ProgVale.

N. The golden rule and the final target

The golden rule for every article

Every lesson should answer at least one of these questions:

• What problem does this solve, at what scale?
• Why does the naive design fail?
• What is the bottleneck?
• What must be consistent, and what can be eventual?
• What happens when this component fails?
• Where does this component live in the architecture?
• Do we actually need it yet?
• What happens when load or requirements change?

That turns the curriculum from a "20 famous systems plus a list of technologies" course into an actual system design training system.

Final curriculum size

Target about 100 to 110 substantial lessons, not 150 or more tiny ones. The first 88 form the structured learning path; the remainder cover failure analysis, operations, advanced design and repeated application. A learner doing 10 hours a week can complete the core path in 7 weeks, and ProgVale can keep resurfacing earlier concepts through spaced practice afterward.

The ultimate competency

New requirement
      ↓
What are the use cases and scale?
      ↓
What are the access patterns?
      ↓
Where is the bottleneck?
      ↓
What must be consistent?
      ↓
What happens when it fails?
      ↓
Do I need this component?
      ↓
What are the trade-offs and costs?
      ↓
How will the system evolve?