F-Pulse Performance Benchmarks — DuckDB Execution on a Single Machine

When evaluating a data pipeline tool, features matter — but performance matters more. A tool that looks great but can't handle your workload is a demo, not a solution.

This post shares real performance numbers from F-Pulse running on a single machine. No cherry-picked numbers, no "up to" claims. Just what you can expect when you run docker compose up -d and start building pipelines.

Test Environment

All benchmarks were run on a commodity machine:

• CPU: Intel i7-12700 (12 cores)
• RAM: 32 GB DDR4
• Storage: NVMe SSD
• OS: Ubuntu 22.04 / Docker
• F-Pulse: v1.0.0 with DuckDB execution engine

No cloud instances, no distributed compute, no Spark cluster. Just one machine.

Benchmark 1: CSV Ingest + Transform + Output

Pipeline: CSV Source (1M rows) → Filter → Transform (SQL) → Aggregate → Parquet Output

Key insight: DuckDB's columnar engine processes 1M rows in ~3 seconds including the full ETL pipeline. For most team workloads (under 10M rows), F-Pulse on a single machine is fast enough that distributed compute adds complexity without benefit.

Benchmark 2: Database-to-Database (PostgreSQL → PostgreSQL)

Pipeline: DB Source (SELECT * with filter) → Transform → Deduplicate → DB Sink (UPSERT)

Key insight: The bottleneck is the database sink (UPSERT), not the pipeline engine. DuckDB processes transforms faster than most databases can write. Batch size tuning on the sink (1000-5000 rows) is the main optimization lever.

Benchmark 3: Concurrent Pipeline Execution

Test: Run N pipelines simultaneously (each: 100K rows, 4 transform nodes)

Key insight: F-Pulse's worker pool handles 25+ concurrent pipelines on a single machine with linear scaling. The per-pipeline time actually decreases with concurrency due to I/O overlap. At 50 concurrent pipelines, CPU is saturated but memory stays manageable.

Benchmark 4: Per-Node Preview Latency

One of F-Pulse's key features is live data preview at every node. How fast is it?

Key insight: Preview is instantaneous (<200ms) for datasets up to 100K rows. Even at 1M rows, the preview loads in under a second. This is what makes "see every step" practical, not just aspirational.

When F-Pulse Is Enough (And When It's Not)

F-Pulse is more than enough for:

• Datasets up to 50M rows on a single machine
• 25+ concurrent pipelines
• Sub-second preview for interactive development
• Daily/hourly batch ETL for most teams
• CDC replication from production databases

You need distributed compute (D-Pulse with Spark/Trino) when:

• Datasets exceed 50M rows regularly
• You need sub-minute processing for multi-GB datasets
• You're joining datasets that don't fit in memory
• You need multi-node parallelism for compliance-driven SLAs

The "Don't Start with Spark" Argument

Most teams reach for Spark too early. Here's why:

1. Spark's minimum overhead is 10-30 seconds just to initialize a job. F-Pulse processes 1M rows in 3 seconds total.

1. Spark requires a cluster — even EMR Serverless or Databricks has cold start latency and cost.

1. Spark's sweet spot is 100M+ rows. Below that, DuckDB on a single machine is faster, cheaper, and simpler.

1. You can always upgrade later. F-Pulse pipelines use an engine-agnostic IR — the same pipeline definition runs on DuckDB locally or Spark/Trino via D-Pulse. No rewrite needed.

The right approach: start with F-Pulse + DuckDB. If you hit scale limits, upgrade to D-Pulse with distributed engines. Don't pay the Spark complexity tax until the data demands it.

Memory Management

DuckDB is efficient with memory, but large datasets still need attention:

• Streaming execution: DuckDB processes data in batches, not all-at-once
• Spill to disk: When memory pressure is high, DuckDB spills intermediate results to disk
• Configurable limits: Set FPULSE_MAX_MEMORY to cap DuckDB's memory usage
• Per-node isolation: Each pipeline step executes independently, releasing memory between steps

How to Run Your Own Benchmarks

F-Pulse includes a sample dataset in data/samples/ for quick testing:

1. Start F-Pulse: docker compose up -d
2. Open the builder: http://localhost:5174
3. Drag a CSV Source → point to data/samples/orders.csv
4. Add transforms (Filter, Aggregate, etc.)
5. Check the execution log for timing at each node

For larger tests, generate data with:

-- In the Transform node's SQL editor:
SELECT
  ROW_NUMBER() OVER () AS id,
  'customer_' || (RANDOM() * 1000)::INT AS customer,
  RANDOM() * 500 AS amount,
  CURRENT_DATE - (RANDOM() * 365)::INT AS order_date
FROM GENERATE_SERIES(1, 1000000)

---

F-Pulse is free and open source. Run your own benchmarks in 3 minutes: Download here.