The plumbing nobody sees
Every beautiful dashboard hides invisible work: fetching data from several systems, cleaning, joining and delivering it — every day, without fail. That is data engineering. When it's done well, nobody notices it exists; when it fails, the board finds out on Monday that the dashboard is showing Thursday's numbers.
Design it or code it? The two schools
There are two ways to build a pipeline. In the visual school, you design the flow in a graphical interface — each box is a step, each arrow is the data's path — and the tool stores that design as metadata that goes into Git like any other code. In the code school, you program the flow in Python. The outcome must be the same: a process that is versioned, scheduled, monitored, and that raises its hand when something goes wrong.
Neither school is "better" — the choice depends on your team, your volume, and on whoever will maintain the pipeline two years from now:
| Approach | Tools | When it makes the most sense |
|---|---|---|
| Visual (low-code) | Apache Hop · Pentaho PDI | Integrations and migrations maintained by a lean team — the flow design is its own documentation |
| Code | Python · Airflow | A dedicated engineering team, highly specific logic, line-by-line automated tests |
| Distributed scale | Apache Spark · Beam | Volume that doesn't fit on one machine — Hop draws the flow, the cluster runs it |
| Managed (cloud) | AWS Glue · Data Fusion · Data Factory | Everything already lives in one cloud and the price works — the concepts are exactly the same |
From Kettle to Apache Hop — we've followed this story from the very start
We have worked with Pentaho Data Integration for over a decade — it powered our SQL Magazine cover story on data migration, and that road took us to teaching a workshop at Pentaho Day 2017 in Curitiba, where we met the creator of Kettle in person. When the platform evolved into Apache Hop, a project created by the original author of Kettle/PDI, we moved right along with it: lightweight metadata that versions cleanly in Git, native container execution, and the option of delegating heavy processing to an Apache Spark or Beam cluster without redrawing the flow. The managed cloud tools are variations on these same ideas — master the concept and you move between them without drama.
This also protects what you already have: legacy PDI flows are reused in the migration to Hop instead of being rewritten from scratch.
What if your shop is all-Microsoft?
Across much of Brazilian retail, corporate systems and the public sector — city governments and e-gov in general — the standard is the Microsoft ecosystem: SQL Server for databases, Azure Data Factory (and Microsoft Fabric) for pipelines, and Azure DevOps for the delivery chain. We are at home in that world: pipelines versioned in Azure Repos, with automated build, test and deployment through Azure DevOps Pipelines — the same CI/CD discipline we apply to Hop and Python, simply using the house tools. And when mixing makes sense, Apache Hop talks to SQL Server and runs in a container on Azure without drama.
When to call us
When reports depend on someone hand-cranking a spreadsheet; when every system tells a different version of the same number; when the "pipeline" is a script only one employee understands — or when it has already failed and nobody noticed in time. We also take over and modernize existing pipelines, including the ones other people built.
What you get
- Pipelines with metadata versioned in Git — no logic locked inside screens
- Scheduling, monitoring and failure alerts — you find out before the board does
- Source-to-target validation: counts and sums checked on every load
- Tool choice justified in writing — visual, code or cloud
- Reuse of what exists: legacy PDI flows migrate into Apache Hop
- Documentation and training so your team takes over without dependency
