The data foundation in detail

How the data foundation is built, layer by layer.

Your data passes through several layers on its way from the systems that record it to the views, reports, and plain-language questions that use it. Each layer does one job and runs on its own software, so one layer's software can be replaced without rebuilding the others.

Newtack operates and monitors every layer.

The layers

What each layer does and what could replace it.

Select a layer to open it.

Where the data comes from

Business applicationsThe software your teams already use. In behavioral health, these are EHR encounters and notes, EHR enrollments and census, care planning, staffing and payroll, and billing and claims.

Operational databasesThe databases where those applications keep their records.

Documents and contractsAgreements, policies, statements, and the spreadsheets people keep alongside them.

Recorded conversationsMeetings and calls, including decisions made in them that were never written down.

Operated by Newtack

Orchestration is the software that decides when each step runs and in what order. The steps include copying data out of your systems, rebuilding the reporting tables, and refreshing the search index for documents. Dagster does this job here.

Dagster keeps a history of every run. It also records which tables each run produced and which tables those were built from. This record is called lineage. When a run fails or a refresh is late, Dagster alerts Newtack's operator.

The schedules and the order of the steps are written in Python, a programming language, and kept in version control. Version control is a system that stores every change to a set of files, with who made it and when. Moving to another orchestrator means rewriting that Python code. The copy jobs, the reporting tables, and the metric definitions stay as they are.

In July 2026, Prefect announced it was acquiring Dagster Labs, the company that makes Dagster. Nothing at this layer had to be rebuilt as a result.

The announcement from Dagster and Prefect

Dagster is released under the Apache 2.0 license. Apache 2.0 is a permissive open-source license. It lets anyone use, change, and share the code, including for commercial work, without paying a license fee.

The project's repository shows code history starting in April 2018. When the repository was read, new changes were being added every day. If the company behind Dagster stopped supporting it, the license would let anyone keep their own copy of the code and maintain it. That kind of copy is called a fork.

Read on 2026-09-10 from the project repository, dagster-io/dagster.

A credential is the username, password, or key that lets software sign in to one of your systems. Newtack's custody layer holds a credential you have issued and hands it to a job only when the job runs. You can revoke it in the source system at any time.

This layer is built on a vault, which is software made to store secrets securely. 1Password, or the secrets manager your team already uses, can hold the credentials. Jobs read them from the vault when they run. Newtack's automation sits on top of the vault. It rotates credentials on a schedule, and it refreshes credentials that a provider would otherwise let expire.

Access ends when you revoke it in the source system.

The automation is Newtack's own software, so this layer adds no new vendor for you to review.

Ingestion is the step that copies data out of your source systems and into the store. dlt does this job. Each copy job is called a pipeline, and each one is written as code.

A pipeline remembers where it stopped. If a run fails, the next run picks up the new records and leaves the ones already copied alone. When a source system adds a field, the pipeline adds it to the table and keeps running.

The pipelines are Python code in version control, and each one writes to tables with fixed names. A different loading tool can write to the same tables. Sources can move to the new tool one at a time while everything above this layer keeps working.

dlt is released under Apache 2.0. It is a Python library with no separate service to run, so it works anywhere the rest of the data foundation runs.

It is a younger project than the components around it. Its job is narrow, so replacing it would be a contained piece of work.

Read on 2026-09-10 from the project repository, dlt-hub/dlt.

Transformation turns the raw copied data into tables organized for reporting. These tables are called models. dbt does this job. The business logic, such as how revenue, census, or utilization is calculated, is written in SQL, the standard language for working with databases. It is kept in version control, so a change to a calculation shows up line by line for someone to review, with the author's name on it.

Every rebuild runs reconciliation tests, which check the new numbers against the source data. If a test fails, the new tables are not published. Yesterday's numbers stay in place until Newtack's operator looks into the failure.

The valuable part of this layer is the models and their tests. They are SQL files with templating, in version control, and other tools that run SQL can read the same files. This layer holds more accumulated work than any other. It also carries the least risk of that work being stranded on one tool.

dbt is released under Apache 2.0, and its repository shows code history starting in March 2016. Fivetran and dbt Labs have completed a merger. dbt Labs has published that dbt Core v2.0 keeps the Apache 2.0 license. It also published that the engine underneath dbt, which is written in the Rust programming language, is being released under the same license.

Read on 2026-09-10 from dbt-labs/dbt-core and from dbt Labs' post on what the merger means.

The reporting tables are stored as Parquet files, organized into Apache Iceberg tables. Parquet is an open, publicly documented file format for table data. Iceberg is an open standard that groups those files into tables that query software can read. The files sit in a cloud account that Newtack operates for you.

A query engine is the software that reads those tables and returns answers to the layer above, which holds the metric definitions. The engine is chosen to fit the size of your data. DuckDB, an analytical database that needs no separate server, fits when the data is small. Postgres, a general-purpose database, fits when the data is already in Postgres and the volume suits a single server. Redshift or a similar data warehouse, a database built for large analytical workloads, fits where the volume calls for one or you have already invested in one.

If you already have a data warehouse that works, it stays in place and becomes the query engine. The engine can be changed later without touching the models or the metric definitions.

Changing the query engine does not move the data. The files stay where they are, and the new engine reads the same tables. This works because Iceberg and Parquet are open formats that every major query engine supports. The layer that holds the metric definitions connects to the new engine. The dashboards and other tools above it keep working as before.

Anyone with access to the storage account can open the files and confirm they are in these formats.

Iceberg and Parquet are both released under Apache 2.0 and run by the Apache Software Foundation. Because an independent foundation runs them, no single vendor can change the license on the format your data is stored in.

Read on 2026-09-10 from the Apache Iceberg and Apache Parquet projects.

Unstructured data is content that does not sit in database tables, such as documents, recordings, and video. The agreement that sets a rate, the policy behind an exception, and the recorded call where a decision was made are all unstructured.

This content comes in on the same schedule as the database records. Audio and video are transcribed as they arrive. A call, a recorded site walkthrough, or a YouTube video becomes text that can be quoted next to the numbers.

Files in Google Drive and SharePoint are read together with their sharing settings. Each file goes into an index, a searchable catalog of the content, along with a record of who may open it. A search is filtered by that record before it returns results.

A common failure in company-wide search is to index every file first and work out who may see what later. This layer records who may open each file at the moment it reads the file.

This layer also keeps a record of the statements the business makes about itself, with the evidence behind each one. Websites and reports take their statements from that record. Newtack keeps the same record for its own published statements.

The original files stay as they are, and everything else is built from them. The index can be rebuilt from the same files, including its vector embeddings. A vector embedding is a list of numbers that captures what a passage means, so a search can find passages with similar meaning even when the words differ.

Changing the search engine means rebuilding the index. The original files are not touched. Postgres with pgvector, an extension that adds vector search to Postgres, stores the embeddings when the amount of material is small. A dedicated vector database stores them when the amount is large.

The connectors and the index are Newtack's own software. Newtack ran them on its own material before any client used them.

Recorded in Newtack's engineering record, accepted on 2026-08-27.

This layer holds the metric definition for every measure, such as revenue or census, and the rules on who may see which data. Cube does this job, and it calls each metric definition a definition. Software of this kind is known as a semantic layer, because it records what each number means.

Each measure has one metric definition, and every change to it is versioned. Every dashboard, embedded view, and question asked in plain language reads that one rule, so they all return the same number.

Access rules are applied by this layer before data reaches a dashboard. Row-level rules limit which records a person can see, such as the locations they manage. Column-level rules hide specific fields. Cube writes these rules into the query itself.

Cube also prepares answers ahead of time for the kinds of questions people ask most, and refreshes them on a schedule. These prepared results are called pre-aggregations. A question from a person or a dashboard is answered from them, so the query engine does not have to read the raw tables each time. This kind of fast analytical querying is called OLAP, short for online analytical processing, and this layer does it.

The metric definitions are configuration files kept in version control. Read on their own, they describe how the business measures itself. They stay useful whichever tool reads them.

The query engine underneath can change without changing the metric definitions. Cube connects to data warehouses, other query engines, and ordinary application databases, so a new query engine means changing one connection at this layer. Nothing above this layer changes.

Moving the metric definitions themselves to a different tool is real work, so this layer changes rarely.

Cube's repository lists Apache 2.0 as the default license, with some packages under MIT. Cube publishes customer stories. One describes how Brex used Cube to build financial analysis into its product for more than 35,000 customers.

Cube's documentation lists the engines it connects to, including Redshift, BigQuery, Snowflake, Databricks, Microsoft Fabric, DuckDB, Athena, Trino, and Postgres. It also describes how AI software, such as a tool that answers plain-language questions, reaches the same metric definitions and access rules through Cube's MCP server and its chat API. MCP, the Model Context Protocol, is a standard way for AI software to connect to tools and data. An API is an interface other software calls to send requests.

Read on 2026-09-10 and 2026-09-24 from cube-js/cube, Cube's published customer stories, and Cube's list of data sources.

Everything that uses the data sits at this layer. The measures, the way tables connect to each other, and the access rules are already settled in the layer below. Adding a new way to use the data is configuration work. It does not start another data project.

The first question in choosing what to build is whether people will use it. A report that someone has to remember to open competes with everything else in their day. The same answer inside an application they already use, or in a message they already read, asks nothing new of them.

Dashboards and reportsThe views leadership uses, refreshed on a schedule.

Embedded in your productViews inside your software that show each customer only its own data, limited on the server before it is sent.

Ask in plain languageQuestions typed in ordinary words and answered from the same metric definitions.

Analyst toolsSQL and notebooks that use the same measures, for questions no report was built to answer. A notebook is a document that mixes code with its results.

Data productsAn API that other systems call, so the same metric definitions serve software as well as people.

This is the most visible layer and the easiest to change. Every tool here reads through the layer that holds the metric definitions, using standard interfaces. Tools that work in SQL use SQL. Other tools use REST, the common way for software to request data over the web. Either way, a new tool starts with the measures and the access rules already in place.

That is why the list of tools can keep growing. A new reporting tool, a plain-language question tool, or a report for your customers can each connect to the same metric definitions. None of them requires changes to the layers below.

Apache Superset is released under Apache 2.0 and run by the Apache Software Foundation. Metabase releases its code under the AGPL, an open-source license that can require you to publish your changes to the code. The code in its enterprise directory is under a commercial license. Because of differences like these, Newtack checks each tool's license against how the tool will be deployed before choosing it.

The tools beyond dashboards are custom-built. They follow the same metric definitions and access rules as everything else, because they read through the same layer.

Read on 2026-09-10 from apache/superset and Metabase's license file.

Who uses the data

The people running the businessViews and scheduled reports, with one metric definition behind each measure.

Your own productViews built into the software your customers already use, each limited to that customer's data.

AnalystsThe same metric definitions through SQL, for questions that go past the views.

Anyone with a questionAnswers to plain-language questions, from the same metric definitions and access rules as a view.

Newtack chooses the components and revisits those choices. The subscription is written around the roles and the results they produce, so it does not commit to a fixed list of products.

What later projects reuse

What each project after the first reuses.

The first project builds the foundation. That means getting data out of the source systems, building the reporting tables, agreeing on the metric definition for each measure, and deciding who may see which records. This is the largest part of the work, and it is done once.

The foundation starts with one view and is sized to fit the first projects.

Each later project uses what already exists. A new dashboard, a view inside your own product, or a question asked in plain language reads the same metric definitions and follows the same access rules.

A question nobody built a report for

An analyst can ask it in SQL, or anyone can ask it in plain language. The answer comes from measures that already agree with each other.

Every view shows the same numbers

A view, a report, and the answer to a plain-language question all use the same metric definition, so they return the same number. Nobody has to decide which of two versions is right.

A metric definition corrected in one place

When a metric definition is wrong, it is corrected in its configuration file, and every view recalculates on the next run. Inconsistent data entry is a separate problem that a rule cannot correct. You can break the figures down by location, program, and clinician to see where it happens. Then you know what to change in the field, and you can see when it has changed.

New tools use the same definitions

A new reporting tool or a tool that answers plain-language questions connects to the same metric definitions through SQL or REST. Adding one does not change the layers below.

Ownership changes

What happened when two of these vendors changed owners.

Two of the companies behind these components changed owners within about two months of each other. Both events are public.

In July 2026, Prefect announced it was acquiring Dagster Labs, including the product, the code, the customer relationships, and much of the team. The announcement says Dagster keeps its name and its open-source license. It says deployments, contracts, and support stay as they are, and that no migration is required. It also says the combined company operates under the Prefect name from August 2026.

Fivetran and dbt Labs completed a merger. dbt Labs published that dbt Core v2.0 is still licensed under Apache 2.0, as it has been since the project began. It also published that the Rust engine underneath dbt is being released under the same license.

Neither event required anything to be rebuilt. Each component from an outside vendor has a permissive license and, where it stores data, an open storage format. The pipelines, models, and metric definitions are kept in version control as files that can move to other tools.

Permissive licenses

The components in the layers that hold your work are released under Apache 2.0. If a vendor stops supporting a component, the license lets anyone keep developing the code.

Open storage formats

Your tables are stored as Parquet files and organized as Apache Iceberg tables. The Apache Software Foundation oversees both formats, and every major query engine can read them.

Work kept in portable form

Transformations are SQL files and metric definitions are configuration files, both kept in version control. When a component is replaced, the work already done carries over to its replacement.

What the subscription covers

Newtack monitors and supports the data foundation.

Newtack chooses the components and manages the production configuration.

Questions

Questions buyers ask.

By default, your data lives in Amazon Web Services, in a cloud account that Newtack operates for you. The data foundation can also run in a cloud account your organization holds. Nothing in the architecture depends on whose account it runs in.

A business intelligence (BI) platform covers every layer with one vendor's product, sold and run as a whole. In the data foundation, each layer is a separate component that works on its own, so one component can be replaced without rebuilding the others.

A single platform is simpler to buy. Separate components take more work to assemble. Newtack does that work and operates the result. It keeps the metric definitions accurate, the access rules enforced, and the views and tools working.

A question asked in plain language reads the data through the layer that holds the metric definitions, the same way a dashboard does. The access rules for the person asking still apply, because that layer applies them before it returns any data. Asking an AI model a question does not open a second route to the data with different rules.

Statements about another company's software come from that project's own repository or published announcement, read in September 2026.

The published sources are the Dagster and Prefect announcement, the Fivetran and dbt Labs post, Cube's customer stories, and the Apache Iceberg project.