One of the toughest challenges in data warehousing today is executing and managing data transformations at scale. The code-first approach adopted by many organizations in recent years is backfiring. Code-first data engineering tools require highly specialized data engineers building bespoke custom scripts that are hard to scale, which ultimately leads to them being unable to keep up with the demands of the business.
We built Coalesce to solve those challenges. Our column-aware architecture enables data teams to build their data warehouse with scalable, reusable, and tested data patterns. Unlike other platforms or tools that require writing ad-hoc custom code, Coalesce uses a stateful column-aware approach, making data transformations accurate, efficient, repeatable, and verified at scale.
What is a data pattern?
A data pattern is a reusable step in a data transformation pipeline that represents a logical transformation. This can include:
- Incremental & processing logic
- Materialization logic
- Deployment logic
Incremental & Logical Processing
While some tools are focused on simple SELECT statements that effectively recreate datasets every time, a fundamental differentiator of the Coalesce platform is that it enables logical incremental processing that fully utilizes the processing power of the data warehouse. While SELECT statements may be simple to start, the reality is that, at scale, data patterns often require incremental materialization, or logical processing that can’t be described efficiently with a SELECT statement, such as typical data patterns like deduplication and change tracking. Data patterns that allow for incremental materialization logic in calculations ensure that data transformations are manageable, accurate, and tested–especially as the project and data sources grow over time.
At Coalesce, we believe that reusable logical transformations and the ability to repeatedly apply them to data is a key data warehouse design consideration.
Materialization logic
Every project uses the bread and butter of table and views, but another consideration for data patterns is that it allows for flexibility in materialization logic.
With data patterns in Coalesce, your team is able to utilize alternative Snowflake object types like external tables, streams and tasks, and do so effectively with column awareness.
This capability, in combination with the logical processing benefits of data patterns, means that your team can apply consistent standards to transformations with external tables, implement streams and tasks as first-class data objects within your pipeline, or implement useful and efficient Snowflake transformation materialization logic like multiple source unions.
Deployment logic
At scale, handling the statefulness of the data warehouse often results in an exponential increase in complexity. Data teams have to track columns, manage changes, renames, changing between views and tables, and managing the deployment process of downstream views appropriately. This can cause deployments to break, leading to constant manual rectification of broken pipelines that only grows as the project scales.
Coalesce packages this deployment logic as part of a data pattern, so the data team doesn’t have to manually perform modifications or write custom code.
Building data patterns efficiently
From above, you can see that using data patterns to build a data warehouse is a scalable and repeatable approach to data transformation.
But in order to make this accessible, it’s also important to consider how these data patterns are built. Without a column-aware architecture, this materialization logic starts to look like custom code scripts and your data project quickly becomes a software project. The design of your data warehouse starts looking more and more like code, with asynchronous SQL statement execution, try catch blocks, error handling logic, exceptions, variables, logging, and so on. While this seems powerful (and it is!), the long-term effects of imperatively describing your data warehouse data patterns with code is that it turns your data team into a software team, your data ops team into a software support team, and your data project into a software project.
We’ve designed Coalesce so that your team never needs to write custom code to build data patterns, while still having all the power of conditionals, iteration, and SQL execution and the ability to automate pipelines at scale with no manual interventions when things go awry.
For serious data projects, Coalesce’s declarative approach to data patterns, enabled by its column-awareness, brings incredible efficiencies to the design, deployment, governance and operation of data warehouse pipelines at scale.
