Governing data products using fitness functions

Governance that runs itself

Data mesh shifts analytical data ownership from a central team to smaller, domain-aligned teams. That distribution creates a governance problem: the traditional model of data stewards manually inspecting assets does not scale and contradicts the autonomy mesh teams are supposed to have. Automation has to carry the enforcement load.

Fitness functions give us a concrete way to do that. A fitness function is a test evaluating how close an implementation is to its stated design objectives, as described in Building Evolutionary Architectures. Applying that idea to data products lets us check architectural characteristics continuously rather than during periodic audits. The goal is to "shift left" on governance — surface issues early in the value stream so teams fix them proactively.

This approach favors three priorities over their alternatives:

• Governance by rule over governance by inspection
• Empowering teams to discover problems over independent audits
• Continuous governance over a dedicated audit phase

The technique is most effective when applied at the level of the data product itself, which acts as the architectural quantum — the smallest valuable, self-contained unit of analytical data — of the mesh. Data products are typically indexed in an organization-wide data catalog that holds rich metadata. Fitness functions can exploit that metadata to verify whether each product meets the architectural characteristics a mesh depends on.

What to assert about a data product

The architectural characteristics laid out in Zhamak Dehghani's Data Mesh: Delivering Data-Driven Value at Scale translate directly into simple assertions. Each can be coded as a check:

Discoverability

Assert that a keyword search by name in the catalog or data product marketplace returns the product in the top-n results.

Addressability

Assert that the data product is accessible via a unique URI.

Self-descriptiveness

Assert that the data product has a proper English description explaining its purpose. Assert for meaningful field-level descriptions.

Security

Assert that access to the data product is blocked for unauthorized users.

Interoperability

• Assert for the existence of business keys, e.g. customer_id, product_id.
• Assert that the data product supplies data via locally agreed, standardized formats such as CSV or Parquet.
• Assert compliance with metadata registry standards like ISO/IEC 11179.

Trustworthiness

Assert for the existence of published SLOs and SLIs, and assert that adherence to those SLOs is good.

Value on its own

Based on the data product name, description, and domain name, assert that the product represents a cohesive information concept in its domain.

Native accessibility

Assert that the data product supports output ports tailored for key personas — for instance, a REST API output port for developers and a SQL output port for data analysts.

Building the fitness functions

These assertions are deliberately simple. Each can be implemented as a standalone test that takes metadata about a data product as its input. With the catalog serving as the source of that metadata, the same set of tests can run against every published product in the mesh.

A minimal fitness function for addressability, for example, would check that a urn or URL field exists and resolves. A trustworthiness check would pull the product's stated SLO targets and compare them with the SLI telemetry recorded since the product's last deployment. Secure access checks would call the product's endpoint with and without credentials, asserting that only the authorized path succeeds.

Running these checks continuously, rather than as a one-off review, means a data product that drifts from its governance baseline is caught the moment the catalog metadata changes or a new port is published. Teams receive the signal locally and can remediate without waiting for a centralized inspection cycle.

Why fitness functions matter for the mesh

The interop promise of a data mesh — network effects across distributed data products — only holds when every product meets a minimum governance bar. Fitness functions are not the whole of automated data governance, and other techniques are outside this discussion. But applied to the product as the architectural quantum, they turn governance from a gate into a feedback loop. The tests give autonomous teams a clear, executable contract for what "good" looks like, and the catalog gives the mesh a central way to observe whether that contract is being kept.

Implementation patterns

Most fitness tests operate on data product metadata stored in the catalog — the discoverability test being an exception. Several implementation routes are available depending on catalog capabilities and testing requirements.

Running assertions within the catalog

Catalogs like Collibra and Datahub support custom logic execution through features such as Collibra workflows and Datahub Metadata Tests. In a recent data mesh implementation using Collibra, we created a custom business asset type called Data Product that simplified fetching all data assets of that type and running assertions via workflows.

Running assertions outside the catalog

When a catalog lacks hook support, or when the available hooks are too restrictive to use preferred testing frameworks, metadata can be pulled via catalog APIs and assertions run in a separate process. For instance, a Trustworthiness fitness test checking for published service level objectives (SLOs) can query the catalog's REST API and use a JSON path library to verify the presence of the relevant fields in the response.

import json
from jsonpath_ng import parse

illustrative_get_dataproduct_response = '''{
  "entity": {
    "urn": "urn:li:dataProduct:marketing_customer360",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Marketing Customer 360",
        "description": "Comprehensive view of customer data for marketing.",
        "domain": "urn:li:domain:marketing",
        "owners": [
          {
            "owner": "urn:li:corpuser:jdoe",
            "type": "DATAOWNER"
          }
        ],
        "uri": "https://example.com/dataProduct/marketing_customer360"
      },
      "dataProductSLOs": {
        "slos": [
          {
            "name": "Completeness",
            "description": "Row count consistency between deployments",
            "target": 0.95
          }
        ]
      }
    }
  }
}'''

def test_existence_of_service_level_objectives():
    response = json.loads(illustrative_get_dataproduct_response)
    jsonpath_expr = parse('$.entity.aspects.dataProductSLOs.slos')
    matches = jsonpath_expr.find(response)

    data_product_name = parse('$.entity.aspects.dataProductProperties.name').find(response)[0].value

    assert matches, "Service Level Objectives are missing for data product : " + data_product_name
    assert matches[0].value, "Service Level Objectives are missing for data product : " + data_product_name

Using LLMs to interpret metadata

Several fitness tests — such as valuable on its own, self descriptiveness, and interoperability compliance with metadata registry standards — require interpreting metadata like field descriptions and job descriptions. Large Language Models (LLMs) are well-suited for this evaluation task.

Using the OpenAI Playground with GPT-4, we can leverage the Function calling feature to extract structured output from these evaluations. The same results are achievable via the OpenAI API. The model is instructed through a get_data_product_fitness function that takes a parameter valuable_on_its_own, an object with two properties: is_fit — a boolean indicating whether the data product is independently valuable — and reason — a text field explaining the rationale.

{
  "name": "get_data_product_fitness",
  "description": "Determine if data product is fit for purpose",
  "strict": false,
  "parameters": {
    "type": "object",
    "required": [],
    "properties": {
      "valuable_on_its_own": {
        "type": "object",
        "properties": {
          "is_fit": {
            "type": "boolean",
            "description": "True if the data product is valuable on it's own, false otherwise"
          },
          "reason": {
            "type": "string",
            "description": "Reason why the data product is valuable on it's own or not"
          }
        },
        "description": "Determines if data product represents a cohesive
                        information concept in its domain. Has value on its own
                        and can be used independent of other data products"
      }
    }
  }
}

When metadata for a customer data product is sent for evaluation, the test for valuable_on_its_own should pass.

User:
{
  "entity": {
    "urn": "urn:li:dataProduct:marketing_customer360",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Marketing Customer 360",
        "description": "Comprehensive view of customer data for marketing.",
        "domain": "urn:li:domain:marketing"
      }
    }
  }
}

Assistant:
 
 get_data_product_fitness({
  "valuable_on_its_own": {
    "is_fit": true,
    "reason": "The 'Marketing Customer 360' data product presumably provides a
               comprehensive view of customer data, which is valuable for marketing
               purposes. It could offer insights into customer behaviors, preferences,
               and engagement with marketing campaigns, all of which can be used to
               improve marketing strategies."
  }
})

In contrast, a Product_Id data product is expected to fail the same test, since a product ID only gains value when joined with product or customer data.

User:
{
  "entity": {
    "urn": "urn:li:dataProduct:product_id",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Product_Id",
        "description": "Table representing product ids per customer",
        "domain": "urn:li:domain:marketing"
      }
    }
  }
}

Assistant: 
 
 get_data_product_fitness({
  "valuable_on_its_own": {
    "is_fit": false,
    "reason": "The 'Product_Id' data product may not be valuable on its own. It simply
               represents product ids per customer and lacks contextual information
               about what those products are. For it to be meaningful, it would
               likely need to be used in conjunction with other data products that
               provide details about the products themselves."
  }
})

Publishing the results

Dashboard tools like Dashing and Dash are well-suited for displaying assertion results in lightweight dashboards; some data catalogs also support custom dashboard creation. Publicly visible dashboards within the organization — showing data products grouped by domain, colored green or red, with drill-down to view failed fitness tests — create a powerful incentive for teams to meet governance standards. They also help data product consumers make informed choices, naturally favoring products that are fit over those that are not.

Fitness as a baseline, not complete governance

Although fitness functions are typically executed centrally within the data platform, accountability for passing tests rests with data product teams. Fitness functions verify adherence to baseline governance standards but do not replace domain-specific requirements. A data product containing clinical trial data, for example, may need additional measures like differential privacy — basic access controls alone are not sufficient.

Still, fitness functions prove their worth in practice. In one client implementation, retrospective evaluation found that over 80% of published data products failed basic fitness tests.

Conclusion

Fitness functions offer an effective governance mechanism for Data Mesh. Given the varying interpretations of the term Data Product, these functions enforce mutually agreed standards, fostering an ecosystem of reusable and interoperable data products. Adherence encourages teams to use the platform's established paved roads, simplifying ongoing maintenance and evolution. Publishing results on internal dashboards strengthens consumer confidence and perception of data quality.