Split out from the original single-file learning.md. Covers streaming a 1.46GB CSV with
BufferedReader and the quoted-comma bug that breaks naive split(",") parsing.


12. CSV Parsing at Scale

For a 1.46GB file, never load it all into memory. Read line by line:

BufferedReader reader = new BufferedReader(new FileReader(filePath))

BufferedReader reads chunks into a buffer — much faster than reading byte by byte.

String line;
boolean isHeader = true;
while ((line = reader.readLine()) != null) {
    if (isHeader) { isHeader = false; continue; } // skip header row
    channel.basicPublish("", queueName, null, line.getBytes());
}

Parsing a CSV line

String[] fields = line.replace("\"", "").split(",");
// "MOMO","MTN","123" → after replace → MOMO,MTN,123 → split → [MOMO, MTN, 123]

Map to object by index (order matches CSV column order):

t.setPaymentTypeId(fields[0]);
t.setSourceId(fields[1]);
t.setAmount(new BigDecimal(fields[16]));

The Quoted Comma Bug — Why split(”,”) Is Not Enough

The approach above has a silent failure mode. Consider a CSV row where one field contains a comma inside quotes:

"MOMO","MTN","TXN001","2024-01-01 00:00:00","ACC","SRC","MOB","TERM","MERCH","PROD","SUB","REF","Smith, John","0244000000","Payment","GHS",50.00,2.50,2024,"MTN","Ghana","TRANSFER","JAN"

After line.replace("\"", ""), the quotes are gone:

MOMO,MTN,TXN001,2024-01-01 00:00:00,ACC,SRC,MOB,TERM,MERCH,PROD,SUB,REF,Smith, John,0244000000,Payment,GHS,50.00,...

Now split(",") sees the comma inside Smith, John as a field separator. Every field from index 13 onwards shifts by one. What was fields[16] (amount = 50.00) is now fields[17] (currency = GHS). When the code calls new BigDecimal("GHS"), Java throws:

Character G is neither a decimal digit number, decimal point, nor "e" notation exponential mark.

The message gets nacked to the DLQ. The record never makes it to the database.

The fix is a proper CSV parser that tracks whether you’re inside quotes:

private String[] splitCsv(String line) {
    List<String> fields = new ArrayList<>();
    StringBuilder current = new StringBuilder();
    boolean inQuotes = false;
    for (char c : line.toCharArray()) {
        if (c == '"') {
            inQuotes = !inQuotes;        // toggle quote state
        } else if (c == ',' && !inQuotes) {
            fields.add(current.toString().trim());
            current.setLength(0);        // reset for next field
        } else {
            current.append(c);
        }
    }
    fields.add(current.toString().trim()); // last field (no trailing comma)
    return fields.toArray(new String[0]);
}

Then parseLine becomes:

String[] fields = splitCsv(line);  // was: line.replace("\"", "").split(",")

Commas inside quotes are now preserved as part of the field value. The field count is always correct regardless of what the data contains.

This is a very common bug in data pipelines. Any time you see split(",") on a CSV line, ask: can any field value contain a comma? If yes — and in real-world data it almost always can (names, addresses, narrations) — the simple split will break silently on those rows.


↑ Back to record-id-generator-java — Learning Notes