NodeFeatureGroup CR Status Not Updating Across Namespaces

Dec 4, 2025 by Alex Johnson 58 views

Introduction

In the realm of Kubernetes, the Node Feature Discovery (NFD) plays a crucial role in detecting and advertising hardware and software capabilities of nodes in a cluster. This information is vital for scheduling workloads that have specific requirements. One key component of NFD is the NodeFeatureGroup custom resource (CR), which allows users to group node features and apply them to nodes based on labels. However, a common issue arises where the status of NodeFeatureGroup CRs created in namespaces other than the one where NFD is deployed does not update correctly. This article delves into the causes of this problem and provides a solution to ensure that the status of NodeFeatureGroup CRs is updated across all namespaces.

Understanding the Issue: NodeFeatureGroup Status Updates

The core of the problem lies in how the NFD master component, responsible for updating the status of NodeFeatureGroup CRs, discovers and processes these resources. By default, the NFD master is configured to monitor NodeFeatureGroup CRs only within its own namespace. This behavior stems from a design choice to limit the scope of the NFD master's operations, potentially for security or performance reasons. However, this can lead to confusion and operational challenges when users create NodeFeatureGroup CRs in other namespaces, expecting their status to be updated based on node feature changes.

To fully grasp the issue, let's break down the key components and processes involved:

Node Feature Discovery (NFD): A Kubernetes add-on that discovers and advertises node features.
NodeFeatureGroup (NFG): A custom resource definition (CRD) that allows grouping node features.
NFD Master: The central component of NFD, responsible for managing node feature discovery and updating the status of NFGs.
Namespaces: Virtual clusters within a Kubernetes cluster, providing isolation and resource management.

The typical workflow involves the following steps:

NFD workers on each node detect and advertise node features.
Users create NodeFeatureGroup CRs to group features and define node selection criteria.
The NFD master processes these NFGs and updates their status based on the discovered node features.
Kubernetes schedulers use the NFG status to make informed scheduling decisions.

When a NodeFeatureGroup CR is created in a namespace different from the NFD master's namespace, the NFD master, by default, does not monitor it. Consequently, the status of the CR remains unchanged, potentially leading to scheduling issues and incorrect workload placement. This is because the NFD master's processing logic, as highlighted in the provided code snippet, explicitly filters for NodeFeatureGroup resources within its own namespace. The code snippet reveals that the getNodeFeatureGroup function, used to retrieve NodeFeatureGroup resources, is called with the NFD master's namespace as a parameter, effectively limiting the scope of the search.

Reproducing the Issue: A Step-by-Step Guide

To demonstrate the problem, follow these steps:

Deploy NFD in the default namespace (or any specific namespace).
Create a new namespace, for example, test-namespace:
```
kubectl create namespace test-namespace
```

Create a NodeFeatureGroup CR in the test-namespace. For example:

apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureGroup
metadata:
  name: example-nfg
  namespace: test-namespace
spec:
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: example.feature.com/present
        operator: Exists
  template:
    labels:
      example.feature.com/nfg: "true"

Observe the status of the NodeFeatureGroup CR:

kubectl get nodefeaturegroup example-nfg -n test-namespace -o yaml

You will notice that the status field of the NodeFeatureGroup CR remains empty or does not reflect the actual state of the nodes in the cluster that match the node selector criteria. This confirms that the NFD master is not processing the NodeFeatureGroup CR in the test-namespace.

Root Cause Analysis: Code Examination

The provided code snippet from the NFD master's source code clearly illustrates the root cause of the issue. The processNodeFeatureGroupUpdateRequest function, responsible for handling updates to NodeFeatureGroup CRs, retrieves resources using the getNodeFeatureGroup function. This function, in turn, uses the NFD master's namespace as a filter when querying the Kubernetes API server. Consequently, only NodeFeatureGroup CRs within the NFD master's namespace are considered for updates.

func (u *updaterPool) processNodeFeatureGroupUpdateRequest(cli nfdclientset.Interface) bool {
	nfgName, quit := u.nfgQueue.Get()
	if quit {
		return false
	}
	defer u.nfgQueue.Done(nfgName)

	nodeFeatureGroupUpdateRequests.Inc()

	// Check if NodeFeatureGroup exists
	var nfg *nfdv1alpha1.NodeFeatureGroup
	var err error
	if nfg, err = getNodeFeatureGroup(cli, u.nfdMaster.namespace, nfgName); apierrors.IsNotFound(err) {
		klog.InfoS("NodeFeatureGroup not found, skip update", "NodeFeatureGroupName", nfgName)
	} else if err := u.nfdMaster.nfdAPIUpdateNodeFeatureGroup(u.nfdMaster.nfdClient, nfg); err != nil {
		if n := u.nfgQueue.NumRequeues(nfgName); n < 15 {
			klog.InfoS("retrying NodeFeatureGroup update", "nodeFeatureGroup", klog.KObj(nfg), "lastError", err)
		} else {
			klog.ErrorS(err, "failed to update NodeFeatureGroup, queueing for retry", "nodeFeatureGroup", klog.KObj(nfg), "lastError", err, "numRetries", n)
		}
		u.nfgQueue.AddRateLimited(nfgName)
		return true
	}

	u.nfgQueue.Forget(nfgName)
	return true
}

This intentional scoping limits the NFD master's operations but creates a blind spot for NodeFeatureGroup CRs in other namespaces. To address this, we need to modify the NFD master's configuration or code to monitor all namespaces.

Solution: Configuring NFD Master to Watch All Namespaces

Option 1: Modifying the Deployment

The most straightforward solution is to modify the NFD master's deployment to watch all namespaces. This can be achieved by setting the --nfg-namespace flag to an empty string (''). This instructs the NFD master to list and watch NodeFeatureGroup resources across all namespaces in the cluster.

To implement this solution, edit the NFD master deployment:

kubectl edit deployment -n <nfd-namespace> nfd-master

Locate the container definition for the NFD master and add or modify the --nfg-namespace flag:

    spec:
      containers:
      - args:
        - --nfg-namespace=''
        - ...
        image: registry.k8s.io/nfd/nfd-master:v0.12.0 # Replace with your NFD version
        name: nfd-master
        ...

Save the changes and Kubernetes will automatically redeploy the NFD master with the new configuration. Once the NFD master is running with this configuration, it will monitor NodeFeatureGroup CRs in all namespaces and update their status accordingly.

Option 2: Patching the Deployment

Alternatively, you can use the kubectl patch command to modify the deployment without manually editing the YAML file. This approach is often preferred for automation and scripting.

kubectl patch deployment nfd-master -n <nfd-namespace> --patch '{"spec": {"template": {"spec": {"containers": [{"name": "nfd-master", "args": ["--nfg-namespace=''"]}]}}}}'

Replace <nfd-namespace> with the actual namespace where NFD is deployed. This command patches the deployment, setting the --nfg-namespace argument to an empty string.

Verification

After applying either of these solutions, verify that the NFD master is correctly monitoring all namespaces by checking the status of NodeFeatureGroup CRs in different namespaces. Create or modify a NodeFeatureGroup CR in a namespace other than the NFD master's namespace and observe whether its status is updated.

Considerations and Best Practices

While configuring the NFD master to watch all namespaces solves the immediate problem, it's important to consider the implications of this change. Monitoring all namespaces increases the workload on the NFD master and potentially impacts its performance, especially in large clusters with numerous namespaces and NodeFeatureGroup CRs. Therefore, it's crucial to monitor the NFD master's resource consumption and adjust its resource limits if necessary.

Security Implications

Granting the NFD master access to all namespaces also has security implications. The NFD master will be able to read all NodeFeatureGroup CRs in the cluster, which might contain sensitive information. Evaluate the security requirements of your environment and implement appropriate access controls if necessary. Kubernetes RBAC (Role-Based Access Control) can be used to restrict the NFD master's access to specific resources or namespaces.

Alternative Approaches

If monitoring all namespaces poses significant performance or security concerns, consider alternative approaches, such as deploying multiple NFD master instances, each responsible for a subset of namespaces. This can distribute the workload and limit the scope of access for each NFD master instance. However, this approach adds complexity to the deployment and management of NFD.

Conclusion: Ensuring Consistent NodeFeatureGroup Status Updates

The issue of NodeFeatureGroup CR status not updating in namespaces other than the NFD master's namespace is a common challenge in Kubernetes environments. Understanding the root cause, as highlighted by the code examination, is crucial for implementing the correct solution. By configuring the NFD master to watch all namespaces, you can ensure that the status of NodeFeatureGroup CRs is consistently updated across the cluster, enabling accurate node feature-based scheduling.

However, it's essential to consider the potential performance and security implications of this change. Monitor the NFD master's resource consumption and implement appropriate access controls to maintain a stable and secure environment. If necessary, explore alternative approaches, such as deploying multiple NFD master instances, to distribute the workload and limit the scope of access.

By carefully evaluating your requirements and implementing the appropriate solution, you can effectively leverage NodeFeatureGroup CRs to enhance workload scheduling and optimize resource utilization in your Kubernetes cluster. For further reading on Kubernetes and related topics, visit the official Kubernetes Documentation.