Compute panels

Self-hosted Comet supports creating custom panels for your experiments using Python. These Python Panels allow you to create custom visualizations and dashboards for your data.

The basic Python Panels run in your browser and have a more limited set of features compared to the optional Comet Compute backed panels.

Comet Compute¶

Comet Compute is a feature which allows you to run Python code on a remote server and display the results in your Comet UI. This feature is useful for running more heavy-weight computations, and provides greater flexibility in terms of the types of visualizations you can create.

In Comet Cloud all Python Panels are backed by Comet Compute, but in self-hosted this feature can be toggled on or off, while still allowing you to use the basic in-browser Python Panels.

Enabling/Disabling Comet Compute¶

Comet Compute is only available in Kubernetes and KAIO deployments.

You can check to see if it is enabled by viewing the pods of your Comet installation. If you see a pod whose name contains python-panels-orchestrator then Comet Compute is enabled on your installation already.

To enable or disable Comet Compute you can explicitly set the following setting in your override-values (cometctl kaio update-config on KAIO):

# ...
comet:
  # ...
  pythonPanels:
    # If already enabled and you wish to disable, set to false:
    enabled: true
# ...

However, this will unfortunately not work out of the box. As a few additional settings are required to make it work.

# ...
ppOrchestrator:
  env:
    PP_HOST_DOMAIN: my-comet.my-company.com
    PP_PROTOCOL: http
    # PP_PROTOCOL: https
  engineValues:
    domain: my-comet.my-company.com
    env:
      ALLOWED_DOMAINS: https://my-comet.my-company.com,http://my-comet.my-company.com,https://www.my-comet.my-company.com,http://www.my-comet.my-company.com
    ppprotocol: http
    # ppprotocol: https
# ...

Be sure to replace all instances of my-comet.my-company.com with the domain of your Comet deployment. And set both instances of PP_PROTOCOL and ppprotocol to https if you are using HTTPS. Pay special attention to the ALLOWED_DOMAINS setting. This is a CORS string and should be set to include the required variations of your domain with the protocol in the URI.

Compute Node Isolation¶

It is also recommended to isolate the compute engines into a separate node pool/group. This can be done by setting the following:

# ...
ppOrchestrator:
  # ...
  engineValues:
    nodeSelector:
      # Set this to whatever label and value you have set on your node pool.
      nodepool: python-panels
  # ...
# ...

Compute Namespace Isolation¶

By default, Python Panels will run in the same namespace as the rest of the Comet application. If you wish to isolate the Python Panels into a separate namespace, you can set the following:

# ...
ppOrchestrator:
  # ...
  env:
    NAMESPACE: python-panels
  # ...
# ...

Specifying/Upgrading Versions¶

Comet Compute Python Panels are versioned separately from the rest of the Comet application. This applies both to the Orchestrator component and the Compute Engine component. You can specify versions set the following:

# ...
ppOrchestrator:
  image:
    tag: x.x.x
  env:
    ENGINE_VERSION: x.x.x
  # ...
# ...

Self-hosting Images¶

Python panels relies on two additional images that are not part of the core Comet product. These images are docker.comet.com/comet-ml/python-panels-orchestrator and docker.comet.com/comet-ml/python-panels-compute-engine.

If you are self-hosting your images, you will need to set the following:

# ...
ppOrchestrator:
  image:
    repository: my-registry.com/python-panels-orchestrator
    tag: latest
  imagePullSecrets: {}
  env:
    ENGINE_VERSION: latest
  engineValues:
    image:
      repository: my-registry.com/python-panels-compute-engine
    imagePullSecrets: {}
    # ...
# ...

Kube DNS discovery¶

Comet Compute requires a DNS discovery service to be running in your Kubernetes cluster. If you are using the default kube-dns service, you do not need to make any changes. If you are using a different service, or have deployed kube-dns with a different name or namespace, you will need to set the following:

# ...
backend:
  # ...
  ppOrchestrator:
    # ...
    resolver:
      ## You can optionally set the ipOverride field if you are using a custom
      ## DNS service that is not deployed in the cluster.
      # ipOverride: 10.0.0.0
      kubeDNSservice:
        # The name of the cluster DNS service.
        name: kube-dns
        # The namespace of the cluster DNS service.
        namespace: kube-system
# ...

Architecture¶

Comet Compute Python Panels are composed of two main components:

Compute Engine: The Compute Engine is responsible for running the Python code that generates the panels. It is a stateless component deployed on demand when a user is expected to request a panel.
Orchestrator: The Orchestrator is responsible for creating and managing the Compute Engines and ensuring that they are running correctly.

Security Considerations¶

Comet Compute involves allowing users to run arbitrary Python code on your server.

This can have security implications, and as with any aspect of running Comet on your own infrastructure, you bear responsibility for ensuring that you take the necessary precautions to secure your installation.

Compute Node Isolation and Compute Namespace Isolation (especially in the absence of Node Isolation, and when paired with Namespace Resource Quotas) are both helpful measures to reduce the risk of heavy or abusive code from starving other workloads on your cluster and causing a Denial of Service, through either honest or malicious usage. However, there are no substitute for Network Isolation.

Network Isolation¶

Code running in the Compute Engine has the potential to access any network resource that the Compute Engine itself has access to, unless properly firewalled.

If there is already a firewall between your users and the environment in which Comet is running, Comet Compute would allow users to bypass that firewall, unless additional measures are taken.

Enabling a Network Policy¶

If your Kubernetes Network Plugin supports it, you can enable a Network Policy to restrict the Compute Engine's access to only the resources it needs to.

Please refer to your Kubernetes Network Plugin's documentation to ensure that Network Policies are supported and enabled.

# ...
ppOrchestrator:
  # ...
  networkPolicy:
    engineEgress:
      ipBlock: 0.0.0.0/0
      except:
        - 10.0.0.0/8
        - 100.64.0.0/10
        - 172.16.0.0/12
        - 192.0.0.0/24
        - 198.18.0.0/15
        - 192.168.0.0/16
# ...

Crafting a Network Policy¶

If enabled, our Helm chart will create two Network Policies attached to the Python Panels Engine pods:

A Default-Deny policy.
A policy that allows access to the network specified in ipBlock, except for any IP that falls within the ip ranges specified in the except list.

Network/IP Ranges are specified in CIDR notation. Where an IPv4 address is followed by a / and the number of bits in the subnet mask. An IP address of all 0's is a wildcard, as is a subnet mask of 0.

In most cases you will want to set the ipBlock to 0.0.0.0/0 and then specify expemptions for your internal network. This will allow the Compute Engine to access the internet, but not your internal network resources. Thus users would not be able to use the Comet Compute Panels to bypass your firewall.

If you do not know the IP ranges of your internal network, the example settings above are a good starting point. They exclude all of the private IP ranges specified in RFC 1918.

If you wish for your Comet Compute Python Panels to access specific resources on your internal network, be sure that the IP of the resource is not covered by the blocks in the except list.

You can craft your own Network Policies (in-addition or instead of those created by the Helm chart) to further allow or restrict access as needed. Simply write a Network Policy YAML file and nest it within the static portion of your override values:

# ...
static:
  # ...
  - kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: my-comet-compute-policy
      namespace: the-namespace-of-your-compute-engine
    spec:
      podSelector:
        matchLabels:
          appType: python-panel-engine
      policyTypes:
        - Egress
      egress:
        - to:
            - ipBlock:
                cidr: 8.8.8.0/24
                except:
                  - 8.8.8.42/32
            - ipBlock:
                cidr: 9.9.9.0/24
                except:
                  - 9.9.9.100/32
# ...

Dec. 17, 2024