Skip to content

Resource Groups

MeshLens uses AWS Resource Groups to read the metadata related to the data products and domains. These resources need to be created in the integrated accounts that were configured in the setup stage.

If you are referencing a data product in an account/region in any of the resource tagging as we outlined in the tagging section, you should have the corresponding resource group in the same account/region. Similarly, if you defined a resource group for a data product, the corresponding domain resource should exist in the same account/region.

It is ok to have multiple of the same data product or domain resource group in different account/region pair. As long as the name of the resource is the same, they will map to the same MeshLens entities. This allows a data product or domain to span multiple account and region.

Note

If there is conflict in the mapping of the data product to a domain, this will be an error and hydration will not complete.

Tip

Example code blocks shown here can be found in the Cloud Formation template that we used before in setting up the Simulation Account.

The following diagram shows grouping of resources based on their tags and tag filters defined in the resource groups.

Resource Groups

Defining a domain

In the example below, we are defining a domain resource group named Covid19Domain which filters resource groups that has tag key/value meshlens:domain=Covid19Domain.

A domain resource group
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
  Covid19DataDomain:
    Type: AWS::ResourceGroups::Group
    Properties:
      Name: Covid19Domain
      Description: Data domain resources that contains Covid19 data products
      ResourceQuery:
        Type: "TAG_FILTERS_1_0"
        Query:
          ResourceTypeFilters:
            - "AWS::ResourceGroups::Group"
          TagFilters:
            - Key: "meshlens:domain"
              Values:
                - "Covid19Domain"
      Tags:
        - Key: "meshlens:domain"
          Value: "self"

Defining a data product

In the example below, we are defining a data product resource group named covid19-world which filters Glue job and crawlers that has tag key/value meshlens:data-product=covid19-world. We also have additional tags to define MeshLens properties for this data product as follows:

  • meshlens-domain: Associated this data product with the corresponding domain resource. The value is the string indicating the name of the domain.
  • meshlens-team: The team that owns the data product.
  • meshlens:data-product: The value is self.
  • meshlens:data-product:quality-threshold: Defines the threshold between healthy and unhealthy data product. The values are between 1 and 10.
  • meshlens:data-product:shelf-life: Defines the number of days before a data product is marked as “expired” from the last update time.
A data product resource group
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
  Covid19WorldDataProduct:
    Type: AWS::ResourceGroups::Group
    Properties:
      Name: covid19-world
      Description: Resources that belong to covid19-world data product
      ResourceQuery:
        Type: "TAG_FILTERS_1_0"
        Query:
          ResourceTypeFilters:
            - "AWS::Glue::Job"
            - "AWS::Glue::Crawler"
          TagFilters:
            - Key: "meshlens:data-product"
              Values:
                - "covid19-world"
      Tags:
        - Key: "meshlens:domain"
          Value: !Ref Covid19DataDomain
        - Key: "meshlens:team"
          Value: "ResearchTeam"
        - Key: "meshlens:data-product"
          Value: "self"
        - Key: "meshlens:data-product:quality-threshold"
          Value: "7"
        - Key: "meshlens:data-product:shelf-life"
          Value: "20"