dataproductpoc-docs

The information captured in this Wiki has been deduced by carrying out a data product proof of concept (PoC)

Introduction

This website documents how to practically create data products within a data mesh.

It is the instantiation of the architectural theory provided by Zhamak Dehgani in her Data Mesh book which is in turn based on thinking formulated with Martin Fowler whilst at ThoughtWorks

It will cover the following topics:-

• What is special about data products
• The data mesh principles and what is a data product
• Data product types
• Data Mesh/Data product architecture
  • Data product & dataset creation architecture
  • Dataset population architecure
  • Dataset consumption architecture
• The data product admin website
• The data marketplace
• The creation of data stores to store data captured from:-
  • Source to an Input Data Port/endpoint
  • Transformed into an abstraction layer
  • Formatted and sent via an Output Data Port/endpoint
• The Enterprise Data Catalogue
• Required documentation & metadata
• The creation of the discovery endpoint which will allow information about the REST API to be discovered i.e. human & machine readable documentation & data dictionary
• The creation of control endpoints which will allow:-
  • Metadata and controls to be posted to a controls input port
  • Audit logs to be pulled from a controls output port
• Improving data quality
• Authentication & authorisation
• How to deploy to a cloud platform
• Data product naming and versioning standards
  • Dataset naming and versioning standards
• Improving Performance
• Capturing metrics and auditing
• Creation of data models in a data modelling tool
• How to create and where to store data lineage diagrams
• Infrastructure

james.m.dey February 2023