Visual ETL with DuckDB: Building Local-First Pipelines in F-Pulse

DuckDB changed what local analytics can feel like. You can scan CSV and Parquet files quickly, join them with database extracts, and produce useful analytical tables without standing up a warehouse for every experiment.

F-Pulse OSS pairs well with that model. It gives DuckDB a visual pipeline surface: sources, transforms, previews, checks, destinations, schedules, and monitoring.

Why DuckDB belongs in visual ETL

DuckDB is excellent when:

• Data fits on one machine or one strong VM
• Files live in CSV, JSON, or Parquet
• Analysts need fast iteration
• The team wants SQL without warehouse overhead
• A pipeline should be portable across laptops, CI, and small servers

F-Pulse adds the operational layer around that work. Instead of keeping a folder of scripts and ad hoc SQL files, you build the flow on a canvas and inspect each node.

Example pipeline

Build a local revenue mart from three inputs:

• orders.parquet
• customers.csv
• A small Postgres table called refunds

The target is a DuckDB table called:

mart_daily_revenue

Step 1: Install F-Pulse OSS

F-Pulse OSS is Apache 2.0 and launched as v1.0.0 on July 20, 2026.

pip install fpulse
fpulse

Open:

http://localhost:8001

Step 2: Add file sources

On the canvas, add:

• A Parquet Source node for orders.parquet
• A CSV Source node for customers.csv

Preview both nodes. Confirm the inferred types before writing any transform logic.

For orders.parquet, expect:

order_id, customer_id, order_ts, region, status, gross_amount

For customers.csv, expect:

customer_id, segment, lifecycle_stage

Step 3: Add the Postgres source

Add a PostgreSQL Source node for refunds:

SELECT
  order_id,
  refunded_amount,
  refunded_at
FROM refunds
WHERE refunded_at >= CURRENT_DATE - INTERVAL '30 days'

F-Pulse OSS includes database, file, storage, SaaS, operations, and vector connectors. DuckDB can sit in the middle as the execution engine for local transformation work.

Step 4: Transform with SQL

Add a SQL Transform node and join the three inputs:

WITH paid_orders AS (
  SELECT
    order_id,
    customer_id,
    DATE_TRUNC('day', order_ts) AS report_date,
    region,
    gross_amount
  FROM orders
  WHERE status = 'paid'
),
refunds_by_order AS (
  SELECT
    order_id,
    SUM(refunded_amount) AS refunded_amount
  FROM refunds
  GROUP BY 1
)
SELECT
  p.report_date,
  p.region,
  COALESCE(c.segment, 'unknown') AS segment,
  COUNT(*) AS order_count,
  SUM(p.gross_amount) AS gross_revenue,
  SUM(COALESCE(r.refunded_amount, 0)) AS refunded_revenue,
  SUM(p.gross_amount - COALESCE(r.refunded_amount, 0)) AS net_revenue
FROM paid_orders p
LEFT JOIN customers c
  ON p.customer_id = c.customer_id
LEFT JOIN refunds_by_order r
  ON p.order_id = r.order_id
GROUP BY 1, 2, 3

Preview the transform. If a column name is wrong, fix it while the graph is still local and cheap to rerun.

Step 5: Add data quality checks

Add checks that prevent bad local marts from becoming trusted outputs:

• report_date is not null
• order_count > 0
• gross_revenue >= 0
• net_revenue >= 0
• refunded_revenue <= gross_revenue

These checks are simple, but they catch the common problems: wrong joins, duplicate rows, type coercion mistakes, and refund over-counting.

Step 6: Write to DuckDB

Add a DuckDB Destination or managed local table sink and write to:

mart_daily_revenue

Use report_date, region, and segment as the upsert key. That lets you rerun the pipeline without duplicating records.

Step 7: Schedule the local job

For a daily refresh:

15 6 * * *

For development, run manually until the output is stable. Then turn on the schedule and add a notification node.

Where this pattern works

DuckDB plus F-Pulse is a good fit for:

• Local analytics marts
• CSV and Parquet cleanup
• Prototype pipelines before warehouse deployment
• Small-team reporting
• Data quality checks before publishing files
• Offline or single-tenant environments

It is less ideal when:

• Data volume requires a distributed warehouse
• The pipeline depends on custom Python libraries
• You need continuous streaming beyond Kafka
• The organization already has a mature dbt and warehouse-only workflow

Add local AI when useful

With Ollama configured, the F-Pulse Copilot can help write SQL against the node schemas:

Create daily net revenue by region and customer segment.
Subtract refunds by order_id.
Keep report_date, region, segment, order_count, gross_revenue, refunded_revenue, net_revenue.

The model drafts the SQL. F-Pulse previews and validates it. That division of labor is what makes local AI useful without making it risky.

Bottom line

DuckDB is fast local SQL. F-Pulse OSS gives it a visual ETL workflow: 66 connectors, live previews, scheduling, monitoring, data quality checks, and Apache 2.0 licensing.

For teams that want local-first analytics without a pile of scripts, the combination is practical immediately.

---

F-Pulse OSS is Apache 2.0 and self-hosted. Install it here or explore the connector catalog.