A complete guide built from real experience: processing a 1.46GB CSV, generating unique 12-digit IDs, pushing records through RabbitMQ, and persisting to MySQL — all in Java.


Record ID Generator — Build Tools, Project Structure & Architecture

Split out from the original single-file learning.md. Covers Java setup, Gradle vs Maven, the
project’s layered/hexagonal architecture, the full pipeline diagram, and a running summary of
changes made across the project. See also 01-java-fundamentals.md and the other domain files
in this folder.


2. Setting Up Java

Check your Java version

java -version

Multiple versions can coexist on the same machine. Enterprise Java uses Java 21 (LTS). LTS = Long Term Support — stable, supported for years.

Set JAVA_HOME

When multiple versions exist, tell your system which one to use:

# Find available versions
update-alternatives --list java
 
# Add to ~/.zshrc
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
 
# Apply
source ~/.zshrc

JAVA_HOME is an environment variable that tells tools (Maven, Gradle, IDEs) where Java lives.


3. Build Tools — Maven vs Gradle

Java projects need a build tool to manage dependencies (external libraries), compile code, and run tests.

MavenGradle
Config filepom.xmlbuild.gradle or build.gradle.kts
LanguageXMLGroovy or Kotlin DSL
Default in enterpriseYesGrowing fast
Android/KotlinNoYes

Gradle with Kotlin DSL (what we used)

mkdir my-project && cd my-project
gradle init --type java-application --dsl kotlin

This generates:

  • app/src/main/java/ — your source code
  • app/src/test/java/ — your tests
  • build.gradle.kts — dependency config
  • gradlew — wrapper script (use this, not gradle directly)

Adding dependencies

In build.gradle.kts:

dependencies {
    implementation("com.rabbitmq:amqp-client:5.21.0")
    implementation("com.mysql:mysql-connector-j:8.3.0")
    implementation("com.zaxxer:HikariCP:5.1.0")
    implementation("org.flywaydb:flyway-core:10.15.0")
    implementation("org.flywaydb:flyway-mysql:10.15.0")
    compileOnly("org.projectlombok:lombok:1.18.32")
    annotationProcessor("org.projectlombok:lombok:1.18.32")
    implementation("org.slf4j:slf4j-api:2.0.13")
    implementation("ch.qos.logback:logback-classic:1.5.6")
}

Running your app

./gradlew run
./gradlew build
./gradlew test

4. Java Project Structure

record-id-generator/
├── app/src/main/java/com/itc/
│   ├── Main.java               # Entry point
│   ├── config/                 # DB, RabbitMQ, Flyway setup
│   ├── model/                  # Data classes (Transaction, Log)
│   ├── repository/             # Database operations
│   ├── service/                # Business logic (ID generation)
│   ├── producer/               # Reads file → pushes to queue
│   └── consumer/               # Reads queue → writes to DB
├── app/src/main/resources/
│   ├── application.properties  # Config
│   └── db/migration/           # SQL migration files
└── app/src/test/java/com/itc/ # Tests

Package naming convention: com.companyname.projectname — reverse domain notation, globally unique.

This structure works fine for a small project. It groups files by technical role (config, model, repository). Every real file in this project has a clear place.

For Scale — Layered / Hexagonal Architecture

As a codebase grows, grouping by technical role breaks down. You end up with 30 files in model/, 30 in repository/, and no way to tell which ones belong to the same feature. The scalable approach is to group by business domain first, then by technical role inside that.

com.itc/
├── domain/                     # Pure business logic — zero framework imports
│   ├── model/                  #   Transaction, Log, and other core types
│   ├── port/                   #   Interfaces your domain depends on
│   │   ├── TransactionStore.java   #     "I need something that can save transactions"
│   │   └── IdGenerator.java        #     "I need something that generates IDs"
│   └── service/                #   Business rules (ID generation, validation)
│       └── IdGeneratorService.java
│
├── application/                # Orchestrates domain — knows about use cases
│   ├── producer/               #   File reading, publishing to queue
│   │   └── FileProducer.java
│   └── consumer/               #   Message processing, batch coordination
│       └── FileConsumer.java
│
├── infrastructure/             # Implements the domain's ports — talks to the outside world
│   ├── db/                     #   MySQL implementation of TransactionStore
│   │   ├── DatabaseConfig.java
│   │   ├── TransactionRepository.java   # implements TransactionStore
│   │   └── LogRepository.java
│   ├── messaging/              #   RabbitMQ wiring
│   │   └── RabbitMQConfig.java
│   └── migration/              #   Flyway
│       └── FlywayConfig.java
│
└── Main.java                   # Wires everything together and starts the app

The key rule: domain never imports infrastructure. IdGeneratorService has no idea MySQL exists. FileConsumer depends on the TransactionStore interface, not TransactionRepository directly. This means you can swap MySQL for PostgreSQL by writing a new TransactionRepository — nothing in domain/ or application/ changes.

This pattern is called Hexagonal Architecture (also known as Ports and Adapters). The “ports” are the interfaces in domain/port/. The “adapters” are the implementations in infrastructure/.

Small project (this one)At scale
Grouped byTechnical roleBusiness domain + role
Fine up to~5 domain conceptsAny size
Changes to MySQL affectOnly repository/Only infrastructure/db/
Can test domain without DBNo — hard to mockYes — domain has no DB dependency


16. The Full Architecture

┌─────────────────────────────────────────────────────┐
│                   Java Application                   │
│                                                      │
│  Thread 1 (Producer)      Thread 2 (Consumer)        │
│  ┌─────────────────┐      ┌──────────────────────┐   │
│  │  FileProducer   │      │   FileConsumer        │   │
│  │                 │      │                       │   │
│  │ Read CSV line   │      │ Receive message       │   │
│  │ Publish to      │      │ Parse CSV line        │   │
│  │ RabbitMQ        │      │ Generate 12-digit ID  │   │
│  └────────┬────────┘      │ Save to MySQL         │   │
│           │               │ Log errors to logs    │   │
└───────────┼───────────────┴──────────┬────────────┘
            │                          │
            ▼                          ▼
      ┌──────────┐              ┌──────────┐
      │ RabbitMQ │─────────────▶│  MySQL   │
      │  Queue   │              │ itc_db   │
      └──────────┘              └──────────┘

17. Key Takeaways

ConceptWhat You Learned
Java setupJAVA_HOME, multiple JDK versions, LTS versions
GradleKotlin DSL, dependencies, ./gradlew run
Project structureLayered architecture: model → repository → service → producer/consumer
Lombok@Data eliminates boilerplate
Propertiesapplication.properties for config, application-local.properties for secrets
HikariCPConnection pooling — never open a new DB connection per operation
FlywaySchema migrations — versioned SQL, auto-applied on startup
Prepared statementsAlways parameterize SQL — never concatenate
RabbitMQDecouples producers and consumers; durable queues survive restarts
Threadsnew Thread(() -> {}).start() — run tasks concurrently
ID generationSecureRandom + digit alphabet + DB unique constraint + retry
CSV at scaleBufferedReader line by line — never load the whole file into memory
LoggingSLF4J + Logback — levels: DEBUG, INFO, WARN, ERROR
Log tableAlways have one for data pipelines — payload, stack trace, timestamp
DockerDev vs prod compose; multi-stage Dockerfile
Static initializerstatic { } runs once at class load — used for shared connections
Try-with-resourcestry (Resource r = ...) — auto-closes anything that implements AutoCloseable
Text blocks"""...""" — multi-line strings without concatenation or escape clutter
Functional interfacesAny interface with one method can be replaced with a lambda
LocalDateTime ↔ JDBCTimestamp.valueOf(localDateTime) bridges Java 8 date types to JDBC
BigDecimal precisionIEEE 754 floating point cannot represent 0.1 exactly — never use double for money
Graceful shutdownthread.join() waits for a thread to finish — missing it causes silent data loss
Streaming vs chunkingBufferedReader already streams — the real bottleneck is one INSERT per message
Batch insertsaddBatch() + executeBatch() — 50–100x faster than one row at a time
ID strategiesTimestamp+random hybrid, UUID v4/v7, Snowflake, ULID, hash-based — each fits different needs
Idempotent pipelineSHA-256 source_hash as a secondary UNIQUE constraint — re-runs skip duplicates safely
Dead Letter QueuebasicNack failed messages instead of acking — lets you replay them after fixing the bug
ObservabilityTrack messages/sec, batch latency, duplicate rate, queue depth — fly with instruments, not blind
PK designGenerated ID = surrogate key (row identity); source_hash = business key (deduplication)


44. Summary of Changes — This Session

The table below is a condensed record of what changed, why, and what you would look for in the code.

ChangeFileWhyWhat to look for
source_hash computation commented outFileConsumer.javaClean data — hash overhead not needed// transaction.setSourceHash(...)
source_hash removed from all INSERTsTransactionRepository.javaMatches above — no hash, no column24 params instead of 25 in setParams
INSERT IGNORE → plain INSERTTransactionRepository.javaNo dedup needed; surfacing real errorsINSERT INTO not INSERT IGNORE INTO
PK collision: parse error, fix one row, retry batchFileConsumer.java1 collision shouldn’t cost 9,999 rowsparseDuplicateId(), loop in flushBatch()
Shared Connection → per-consumer ConnectionRabbitMQConfig.javaOne reader thread was a bottleneckfactory.newConnection() inside createChannel()
NUM_CONSUMERS 6 → 10Main.javaMore parallel inserts — 10 ran stablyprivate static final int NUM_CONSUMERS = 10
HikariCP pool 20 → 30DatabaseConfig.javaCover 10 consumers with headroomconfig.setMaximumPoolSize(30)
batchLock + synchronizedFileConsumer.javaScheduler + callback on different threadsprivate final Object batchLock
ScheduledExecutorService schedulerFileConsumer.javaTail-end partial batches never flushedExecutors.newSingleThreadScheduledExecutor()
basicQos(BATCH_SIZE) → basicQos(BATCH_SIZE * 2)FileConsumer.javaEliminate idle gap between insertschannel.basicQos(BATCH_SIZE * 2)
Thread.currentThread().join() in drain modeMain.javaJVM was exiting before consumers finishedIn the else branch of if (args.length > 0)
declareQueues moved to Main.javaMain.java / FileConsumer.java10 threads simultaneously declaring the same queue caused a silent deadlocktry (Channel setup = ...) { declareQueues(setup); } before consumer loop
splitCsv() replaces split(",")FileConsumer.javaQuoted fields containing commas shifted all column indicessplitCsv(line) in parseLine()
Shovel plugin for DLQ replayRabbitMQ containerMove failed messages back to main queue after fixing the bugdocker exec <container> rabbitmq-plugins enable rabbitmq_shovel rabbitmq_shovel_management

Built during SIWES @ ITC, Accra — June 2026

↑ Back to record-id-generator-java — Learning Notes