Source: record-id-generator-java/learning/07-id-generation-and-idempotency.md.

Generating unique IDs

public class IdGeneratorService {
    private static final String DIGITS = "0123456789";
    private final SecureRandom random = new SecureRandom();   // cryptographically strong — unpredictable, unlike Random
 
    public String generate() {
        StringBuilder sb = new StringBuilder(12);
        for (int i = 0; i < 12; i++) sb.append(DIGITS.charAt(random.nextInt(DIGITS.length())));
        return sb.toString();
    }
}

The birthday paradox — when collisions actually become likely

With k possible values and n generated IDs, the expected number of collisions across the whole set is approximately n² / (2k). The counterintuitive part: collisions become non-negligible once n approaches √k, not k itself.

For 12 random decimal digits (k = 10¹²), the 50%-per-insert collision threshold is reached around √(10¹²) ≈ 1,000,000 rows — comfortably past a typical single load of a few million rows, where the expected collision count stays in the low tens, cheaply absorbed by a retry loop:

int attempts = 0;
while (attempts < 3) {
    try {
        entity.setId(idGenerator.generate());
        repository.save(entity);
        return;
    } catch (SQLIntegrityConstraintViolationException e) {
        attempts++;   // retry with a freshly generated ID
    }
}

In-memory dedup for bulk loads with no DB round-trip (used when the unique index is dropped for load speed — see 03-batch-processing-and-performance): track generated IDs in a set before they ever reach the DB.

// Single-threaded
Set<Long> usedIds = new HashSet<>(8_000_000);
String id = idGenerator.generate();
while (!usedIds.add(Long.parseLong(id))) id = idGenerator.generate();   // add() returns false if already present
 
// Shared across multiple consumer threads
Set<Long> usedIds = ConcurrentHashMap.newKeySet(8_000_000);   // thread-safe add() — see 04-concurrency-and-the-jvm

At scale, a HashSet<Long> of several million boxed longs costs hundreds of MB. A Bloom filter (e.g. Guava) trades exactness for ~50x less memory — it can false-positive (says “seen” when it hasn’t, triggering a harmless extra regeneration) but never false-negatives, which is exactly the failure mode you want for ID dedup.

Comparing ID strategies

StrategyCollision spaceSortableNotes
Pure random digitsLarge (10ⁿ)NoSimple; fine below the √k threshold
Timestamp prefix + random suffixSmall — bounded by the suffix widthYesSorts chronologically, but the random part must be wide enough for daily/monthly volume or collisions start almost immediately
UUID v42¹²⁸ — negligibleNoUniversal, zero coordination — but fully random, so as an indexed/PK column it causes the clustered-index page-split problem in 01-jdbc-and-data-modeling
UUID v72¹²⁸ — negligibleYes (ms timestamp prefix)Same universality as v4, but new rows insert near the “right end” of a B-tree index instead of at random positions — the fix for v4’s index-fragmentation cost
Snowflake (Twitter)[41-bit ms timestamp][10-bit worker id][12-bit sequence] in a 64-bit longYesCompact, numeric, coordination-free across machines via the worker-ID bits; 4096 IDs/ms/machine
ULID48-bit timestamp + 80-bit random, Base32YesURL-safe, same 128 bits as a UUID
Hash-based (deterministic)Depends on truncation lengthNoSame source data always produces the same ID — free deduplication with no retry logic, at the cost of truncation collision risk if hashed and cut short

The general lesson in choosing between these: a timestamp-prefixed ID looks like a strict improvement (sortable, meaningful) but the math often runs backwards for high-volume ingestion — a narrow random suffix (say, 4 digits for daily precision) hits its own birthday-paradox threshold at a few hundred rows, while 12 fully random digits doesn’t hit it until roughly a million. Sortability and low collision risk are in tension, and the right tradeoff depends entirely on expected volume per unit of time precision.

Idempotent pipelines

A pipeline is idempotent if running it once or a hundred times against the same input produces the same end state — critical because pipelines fail and get re-run.

Making a random-PK design idempotent: since the surrogate key changes on every attempt (it’s random), it can’t detect a re-run by itself. Add a business key — a deterministic fingerprint of the fields that identify a source record — as a secondary unique constraint:

private String computeHash(Transaction t) {
    String input = t.getSourceTransId() + "|" + t.getSourceId() + "|" + t.getSourceDateCreated();
    byte[] hash = MessageDigest.getInstance("SHA-256").digest(input.getBytes(StandardCharsets.UTF_8));
    return HexFormat.of().formatHex(hash);
}
ALTER TABLE transactions ADD COLUMN source_hash VARCHAR(64) NOT NULL, ADD UNIQUE INDEX ux_source_hash (source_hash);

On re-run, the same source record produces the same hash, and the unique index rejects the duplicate insert — a violation that’s expected and handled as a no-op, not an error:

} catch (SQLIntegrityConstraintViolationException e) {
    if (e.getMessage().contains("ux_source_hash")) {
        log.info("Skipping duplicate: {}", t.getSourceTransId());   // already processed — not an error
    } else {
        attempts++;   // an actual PK collision — regenerate and retry
    }
}

MySQL can also absorb this at the SQL level instead of the application level: INSERT IGNORE silently skips a row that would violate a unique constraint (no exception thrown, batch continues), while ON DUPLICATE KEY UPDATE overwrites instead of skipping. INSERT IGNORE also silently swallows other kinds of errors (type mismatches, etc.), which can mask real bugs — a deliberate tradeoff, not a free lunch.

Seen in the wild

↑ Back to 06 — Applied Systems