Clusterpedia v0.9.0-beta.2

Merge pull request #802 from Iceber/fix_hack_script

fix hack/verify-forked-code.sh

Clusterpedia v0.9.0-beta.1

Notable Changes since v0.8.0

Global Features

Associated Storage and Querying of Resource-Specific Events

The related PRs: (@Iceber, #783, #785, #786, #788, #787)

This feature implements the basic functionality for storing and retrieving event associations. It will be continuously iterated on based on user needs in the future.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster-example
spec:
  kubeconfig: "**base64**"
  syncResources:
    - group: ""
      resources:
      - "*"
      eventsInvolvedResources:
      - "pods"
      - "nodes"

Use spec.syncResources.[].eventsInvolvedResources to specify the resources whose events need to be synchronized.
Note: If eventsInvolvedResources includes resources that have already been synchronized, you'll need to restart the clustersynchro-manager. This behavior will be optimized in the future based on user needs.

Get Events

Use the search label search.clusterpedia.io/inject-events=true to request that the returned data includes events.
The events will be injected into the shadow.clusterpedia.io/events annotation.

Note: The events are not currently sorted. Event filtering and sorting may be added in the future.

$ kubectl --cluster kpanda-global -n kpanda-system get po -l "search.clusterpedia.io/inject-events=true" -o yaml
...
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      shadow.clusterpedia.io/cluster-name: kpanda-global
      shadow.clusterpedia.io/events: '[{"kind":"Event","apiVersion":"v1","metadata":{"name":"kpanda-clusterpedia-controller-manager-67758bb46d-mpl49.1843f13522e37ad9","namespace":"kpanda-system","uid":"2220a44a-5414-4698-bc89-16ae411f5dc1","resourceVersion":"23444068","creationTimestamp":"2025-06-23T01:49:12Z"},"involvedObject":{"kind":"Pod","namespace":"kpanda-system","name":"kpanda-clusterpedia-controller-manager-67758bb46d-mpl49","uid":"1918b3fb-cf36-4a2f-8f35-2e105210c38c","apiVersion":"v1","resourceVersion":"7019","fieldPath":"spec.containers{kpanda-clusterpedia-controller-manager}"},"reason":"Created","message":"Created container: kpanda-clusterpedia-controller-manager","source":{"component":"kubelet","host":"g-master68"},"firstTimestamp":"2025-05-29T08:11:30Z","lastTimestamp":"2025-06-23T02:57:05Z","count":393,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"g-master68"},{"kind":"Event","apiVersion":"v1","metadata":{"name":"kpanda-clusterpedia-controller-manager-67758bb46d-mpl49.1843f215c90791e6","namespace":"kpanda-system","uid":"3e1417e9-68df-4b2a-947e-f8a37178429b","resourceVersion":"23444060","creationTimestamp":"2025-06-23T01:49:12Z"},"involvedObject":{"kind":"Pod","namespace":"kpanda-system","name":"kpanda-clusterpedia-controller-manager-67758bb46d-mpl49","uid":"1918b3fb-cf36-4a2f-8f35-2e105210c38c","apiVersion":"v1","resourceVersion":"7019","fieldPath":"spec.containers{kpanda-clusterpedia-controller-manager}"},"reason":"Pulled","message":"Container ....'
...

Historical Events and Cleanup Policy

Associated stored events will not be deleted when the corresponding events in the cluster are cleaned up. This means historical events will be retained permanently until the related resource itself is deleted.

Get Cluster Authentication Info from Secrets

The related PRs: (@Iceber @scydas, #753, #747)

You need to enable the feature gate --feature-gates=ClusterAuthenticationFromSecret=true for both clusterpedia-apiserver and clustersynchro-manager.

Then, configure the PediaCluster and its corresponding authentication Secrets.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster-example
spec:
  apiserver: "https://cluster-example.io:8080"
  authenticationFrom:
    kubeconfig:
      name: "cluster-example-auth-kubeconfig"
      key: "kubeconfig"
    ca:
      name: "cluster-example-auth"
      key: "ca.crt"
    cert:
      name: "cluster-example-auth"
      key: "client.crt"
    key:
      name: "cluster-example-auth"
      key: "client.key"
    token:
      name: "cluster-example-auth"
      key: "token"
  syncResources:
    - group: ""
      resources:
      - "*"
---
apiVersion: v1
data:
  ca.crt: LS0tLS1...base64
  client.crt: LS0tLS1...base64
  client.key: LS0tLS1...base64
kind: Secret
metadata:
  name: cluster-example-cert
  namespace: clusterpedia-system
type: Opaque

Note: You don’t need to configure every field in spec.authenticationFrom. Just provide a valid combination, such as: 1. Only kubeconfig 2. ca + cert + key 3. ca + token

Authentication Priority:

1. spec.kubeconfig > spec.cert & spec.key&spec.token > spec.authenticationFrom
2. spec.authenticationFrom: .kubeconfig > .cert & .key & .token

The namespace of the cluster authentication secrets must match the namespace where the Clusterpedia components are deployed.

Clusterpedia Apiserver

• apiserver: remove admission middleware (@Iceber, #674)

Request Forwarding

In this version, Clusterpedia supports request forwarding to member clusters. This feature is divided into two categories:

1. Proxy specific subresource requests
2. Pass through all requests directly

Stricter authentication configurations have also been introduced for this feature.

To enable it, you need to turn on the AllowProxyRequestToClusters feature gate and set the appropriate flags in the Clusterpedia API server.

$ ./bin/apiserver --help
Resource server flags:

      --allow-forward-unsync-resource-request
                Allow forwarding requests for unsynchronized resource types.By default, only requests for resource types configured in PediaCluster can be forwarded.
      --allow-pediacluster-config-for-proxy-request
                Allow proxy requests to use the cluster configuration from PediaCluster when authentication information cannot be got from the header.
      --allowed-proxy-subresources strings
                List of subresources that support proxying requests to the specified cluster, formatted as '[resource/subresource],[subresource],...'. Supported proxy subresources include
                "services/proxy,pods/proxy,pods/log,pods/exec,pods/attach,pods/portfowrd,nodes/proxy".
      --enable-proxy-path-for-forward-request
                Add a '/proxy' path in the API to proxy any request.

Proxying Subresource Requests

The related PRs: (@Iceber @scydas, #715, #717, #723 , #719)

Enable this by setting the --allowed-proxy-subresources flag in the Clusterpedia API server.

$ # Enable the proxy subresources for all resources and the exec subresources of the pods.
$ ./bin/apiserver --allowed-proxy-subresources "pods/exec,proxy"

Currently supported subresource requests include:

• pods/log
• pods/exec
• pods/attach
• pods/portfowrd
• pods/proxy
• nodes/proxy
• services/proxy

The usage is consistent with native Kubernetes behavior. You can use kubectl to make calls. For example:

$ kubectl --cluster cluster-example logs nginx
$ kubectl --cluster cluster-example exec -ti nginx bash

Request Forwarding

The related PRs: (@Iceber @scydas, #716, #741, #748, #752)

Clusterpedia supports three ways to forward requests directly to member clusters:

1. Enable the --enable-proxy-path-for-forward-request flag. The API server will support proxy-style paths like:
   /apis/clusterpedia.io/v1beta1/resources/clusters//proxy/
2. Use the HTTP header: X-CLUSTERPEDIA-FORWARD: true
3. For list requests, use the search label —— search.clusterpedia.io/forward to enable forwarding.

# kubeconfig
- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:8443/apis/clusterpedia.io/v1beta1/resources/clusters/cluster-example/
  name: cluster-example
- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:8443/apis/clusterpedia.io/v1beta1/resources/clusters/cluster-example/proxy
  name: cluster-example-proxy

$ kubectl --cluster cluster-example-proxy get po -A

$ kubectl --cluster cluster-example get po -l search.clusterpedia.io/forward=true

$ curl -v -k --cert-type P12 --cert client.p12:password  -H "X-CLUSTERPEDIA-FORWARD: true" "https://localhost:8443/apis/clusterpedia.io/v1beta1/resources/clusters/cluster-example/api/v1/namespaces/clusterpedia-system/pods"

You can enable a more relaxed mode by setting the --allow-forward-unsync-resource-request flag to allow forwarding requests for unsynchronized resources.

Authentication for Proxy Requests

• apiserver: set proxy auth info via request header (@Iceber, #727)

By default, proxy requests require authentication headers to be passed in the request for the target cluster. These headers include the necessary credentials.

1. X-Clusterpedia-Proxy-CA
2. X-Clusterpedia-Proxy-Token
3. X-Clusterpedia-Proxy-Client-Cert
4. X-Clusterpedi...

Clusterpedia v0.9.0-beta.0

What's Changed

• build(deps): bump golang from 1.21.5 to 1.21.6 by @dependabot in #640
• Change the storage plugin name to sample-storage by @mayur-tolexo in #641
• feat: add pprof router by @jiuker in #646
• fix otel trace handler WithTracing, use another func by @KubeKyrie in #638
• remove useless code and fix typo error by @googs1025 in #653
• build(deps): bump golangci/golangci-lint-action from 3 to 4 by @dependabot in #652
• support cluster filter when use memory storage by @huiwq1990 in #649
• build(deps): bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #658
• fix the version order of 'discovery.k8s.io' by @Iceber in #659
• build(deps): bump golangci/golangci-lint-action from 5 to 6 by @dependabot in #664
• use a separate action to validate the forked code by @Iceber in #669
• Bump kubernetes to v1.30.2 by @Iceber in #668
• update hack/.import-aliases by @Iceber in #670
• golangci-lint: remove deprecated linters by @Iceber in #671
• install storagemigration group to LegacyResourceScheme by @Iceber in #672
• apiserver: remove admission middleware by @Iceber in #674
• apiserver: unlock the RemainingItemCount feature gate by @Iceber in #675
• e2e: add test for several control plane versions by @Iceber in #676
• Use actions to install kwokctl by @wzshiming in #596
• fix the memory version returned by storageconfig factory by @Iceber in #679
• move discovery, scheme and informer to pkg/runtime by @Iceber in #682
• add pkg/runtime/resourceconfig.ResourceConfig by @Iceber in #678
• bump alpine from 3.18.5 to 3.18.8 by @Iceber in #688
• bump golang from 1.22.4 to 1.22.6 by @Iceber in #689
• dependabot: ignore minor version for golang and alpine images by @Iceber in #690
• Bump kubernetes to v1.31.0 by @Iceber in #691
• bump controller-runtime and controller-tools by @Iceber in #692
• metrics: use legacyregister instead of new registry by @Iceber in #694
• kustomize: remove traces config by @Iceber in #693
• metrics: register clusterpedia and kubernetes build info by @Iceber in #695
• metrics: add component-base/metrics.Options by @scydas in #696
• support for storage to implement custom resource synchro by @Iceber in #677
• build(deps): bump alpine from 3.18.8 to 3.18.9 by @dependabot in #698
• build(deps): bump golang from 1.22.6 to 1.22.7 by @dependabot in #697
• build(deps): bump golang from 1.22.7 to 1.22.8 by @dependabot in #701
• bump kubernetes to v1.31.2 by @Iceber in #704
• test: check the clusterpedia apiserver before test cases by @Iceber in #705
• build(deps): bump golang from 1.22.8 to 1.22.9 by @dependabot in #702
• memorystorage: fix deleting cluster info when cleaning a resource by @Iceber in #706
• add kubeVela and vCluster in Readme by @pacoxu in #707
• add memory storage layer v2 by @Iceber in #708
• internalstorage: add gorm database metrics by @Iceber in #711
• build(deps): bump golang from 1.22.9 to 1.22.10 by @dependabot in #710
• internalstorage: add metrics config by @Iceber in #712
• internalstorage: change the prefix of the db stats metrics by @scydas in #713
• internalstorage: add trace by @scydas in #714
• support pod subresources by @Iceber in #715
• apiserver: Adding RemoteProxyREST for pass-through any requests by @Iceber in #716
• apiserver: add proxy the proxy subresources of pod/node/service by @scydas in #717
• apiserver: fix covert connect methods to kube verbs by @scydas in #719
• build(deps): bump alpine from 3.18.9 to 3.18.10 by @dependabot in #720
• build(deps): bump alpine from 3.18.10 to 3.18.11 by @dependabot in #721
• build(deps): bump golang from 1.22.10 to 1.22.11 by @dependabot in #722
• ci: bump upload artifact to v4 by @Iceber in #726
• build(deps): bump golang from 1.22.11 to 1.22.12 by @dependabot in #725
• Support shutting down storage factory by @learner0810 in #724
• apiserver: add enable subresource options by @Iceber in #723
• apiserver: set proxy auth info via request header by @Iceber in #727
• clustersynchro: add resource synchro metrics by @Iceber in #718
• build(deps): bump alpine from 3.18.11 to 3.18.12 by @dependabot in #728
• bump go.uber.org/atomic from 1.10.0 to 1.11.0 by @Iceber in #737
• apiserver: forward request with prefix path or header by @Iceber in #741
• fix shutdown the storage factory by @scydas in #749
• apiserver: forward list request with labelselector by @scydas in #748
• apiserver: fix the query param in proxy requests by @scydas in #750
• support referencing cluster authentication info from secrets by @Iceber in #747
• apiserver: add feature gate AllowProxyRequestToClusters by @scydas in #752
• add feature gate ClusterAuthenticationFromSecret by @scydas in #753
• bump github.com/stretchr/testify from 1.9.0 to 1.10.0 by @Iceber in #757
• bump github.com/jinzhu/configor from 1.2.1 to 1.2.2 by @Iceber in #758
• bump gorm.io dependencies and fix conflicts by @Iceber in #767
• bump gorm.io/driver/sqlite to v1.5.7 by @Iceber in #769
• bump github.com/DATA-DOG/go-sqlmock to v1.5.2 by @Iceber in #768
• bump github.com/jackc/pgx/v5 from 5.5.5 to 5.7.2 by @Iceber in #772
• kubeapiserver: allow using the storage version in request handling by @Iceber in #771
• bump golangci-lint to v2 by @Iceber in #784
• record events for resources by @Iceber in #783
• internalstorage: add methods for getting resource events by @Iceber in #785
• add search label to inject events annotation by @Iceber in #786
• EventsSynchro: handle orphan events by @Iceber in #788
• EventsSynchro: delete cached event key by @Iceber in #787
• kubeapiserver: fix typo by @Iceber in #790
• clustersynchro: check the fields of resourceversions from storage by @Iceber in https://github.com/cl...

Clusterpedia v0.9.0-alpha.2

What's Changed

• bump gorm.io dependencies and fix conflicts by @Iceber in #767
• bump gorm.io/driver/sqlite to v1.5.7 by @Iceber in #769
• bump github.com/DATA-DOG/go-sqlmock to v1.5.2 by @Iceber in #768
• bump github.com/jackc/pgx/v5 from 5.5.5 to 5.7.2 by @Iceber in #772
• kubeapiserver: allow using the storage version in request handling by @Iceber in #771
• bump golangci-lint to v2 by @Iceber in #784
• record events for resources by @Iceber in #783
• internalstorage: add methods for getting resource events by @Iceber in #785
• add search label to inject events annotation by @Iceber in #786
• EventsSynchro: handle orphan events by @Iceber in #788
• EventsSynchro: delete cached event key by @Iceber in #787

Full Changelog: v0.9.0-alpha.1...v0.9.0-alpha.2

Clusterpedia v0.9.0-alpha.1

What's Changed

• build(deps): bump golang from 1.21.5 to 1.21.6 by @dependabot in #640
• Change the storage plugin name to sample-storage by @mayur-tolexo in #641
• feat: add pprof router by @jiuker in #646
• fix otel trace handler WithTracing, use another func by @KubeKyrie in #638
• remove useless code and fix typo error by @googs1025 in #653
• build(deps): bump golangci/golangci-lint-action from 3 to 4 by @dependabot in #652
• support cluster filter when use memory storage by @huiwq1990 in #649
• build(deps): bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #658
• fix the version order of 'discovery.k8s.io' by @Iceber in #659
• build(deps): bump golangci/golangci-lint-action from 5 to 6 by @dependabot in #664
• use a separate action to validate the forked code by @Iceber in #669
• Bump kubernetes to v1.30.2 by @Iceber in #668
• update hack/.import-aliases by @Iceber in #670
• golangci-lint: remove deprecated linters by @Iceber in #671
• install storagemigration group to LegacyResourceScheme by @Iceber in #672
• apiserver: remove admission middleware by @Iceber in #674
• apiserver: unlock the RemainingItemCount feature gate by @Iceber in #675
• e2e: add test for several control plane versions by @Iceber in #676
• Use actions to install kwokctl by @wzshiming in #596
• fix the memory version returned by storageconfig factory by @Iceber in #679
• move discovery, scheme and informer to pkg/runtime by @Iceber in #682
• add pkg/runtime/resourceconfig.ResourceConfig by @Iceber in #678
• bump alpine from 3.18.5 to 3.18.8 by @Iceber in #688
• bump golang from 1.22.4 to 1.22.6 by @Iceber in #689
• dependabot: ignore minor version for golang and alpine images by @Iceber in #690
• Bump kubernetes to v1.31.0 by @Iceber in #691
• bump controller-runtime and controller-tools by @Iceber in #692
• metrics: use legacyregister instead of new registry by @Iceber in #694
• kustomize: remove traces config by @Iceber in #693
• metrics: register clusterpedia and kubernetes build info by @Iceber in #695
• metrics: add component-base/metrics.Options by @scydas in #696
• support for storage to implement custom resource synchro by @Iceber in #677
• build(deps): bump alpine from 3.18.8 to 3.18.9 by @dependabot in #698
• build(deps): bump golang from 1.22.6 to 1.22.7 by @dependabot in #697
• build(deps): bump golang from 1.22.7 to 1.22.8 by @dependabot in #701
• bump kubernetes to v1.31.2 by @Iceber in #704
• test: check the clusterpedia apiserver before test cases by @Iceber in #705
• build(deps): bump golang from 1.22.8 to 1.22.9 by @dependabot in #702
• memorystorage: fix deleting cluster info when cleaning a resource by @Iceber in #706
• add kubeVela and vCluster in Readme by @pacoxu in #707
• add memory storage layer v2 by @Iceber in #708
• internalstorage: add gorm database metrics by @Iceber in #711
• build(deps): bump golang from 1.22.9 to 1.22.10 by @dependabot in #710
• internalstorage: add metrics config by @Iceber in #712
• internalstorage: change the prefix of the db stats metrics by @scydas in #713
• internalstorage: add trace by @scydas in #714
• support pod subresources by @Iceber in #715
• apiserver: Adding RemoteProxyREST for pass-through any requests by @Iceber in #716
• apiserver: add proxy the proxy subresources of pod/node/service by @scydas in #717
• apiserver: fix covert connect methods to kube verbs by @scydas in #719
• build(deps): bump alpine from 3.18.9 to 3.18.10 by @dependabot in #720
• build(deps): bump alpine from 3.18.10 to 3.18.11 by @dependabot in #721
• build(deps): bump golang from 1.22.10 to 1.22.11 by @dependabot in #722
• ci: bump upload artifact to v4 by @Iceber in #726
• build(deps): bump golang from 1.22.11 to 1.22.12 by @dependabot in #725
• Support shutting down storage factory by @learner0810 in #724
• apiserver: add enable subresource options by @Iceber in #723
• apiserver: set proxy auth info via request header by @Iceber in #727
• clustersynchro: add resource synchro metrics by @Iceber in #718
• build(deps): bump alpine from 3.18.11 to 3.18.12 by @dependabot in #728
• bump go.uber.org/atomic from 1.10.0 to 1.11.0 by @Iceber in #737
• apiserver: forward request with prefix path or header by @Iceber in #741
• fix shutdown the storage factory by @scydas in #749
• apiserver: forward list request with labelselector by @scydas in #748
• apiserver: fix the query param in proxy requests by @scydas in #750
• support referencing cluster authentication info from secrets by @Iceber in #747
• apiserver: add feature gate AllowProxyRequestToClusters by @scydas in #752
• add feature gate ClusterAuthenticationFromSecret by @scydas in #753
• bump github.com/stretchr/testify from 1.9.0 to 1.10.0 by @Iceber in #757
• bump github.com/jinzhu/configor from 1.2.1 to 1.2.2 by @Iceber in #758

New Contributors

• @mayur-tolexo made their first contribution in #641
• @jiuker made their first contribution in #646
• @googs1025 made their first contribution in #653
• @pacoxu made their first contribution in #707
• @learner0810 made their first contribution in #724

Full Changelog: v0.8.0...v0.9.0-alpha.1

Clusterpedia v0.8.1

What's Changed

• [release/0.8] fix the version order of 'discovery.k8s.io' by @Iceber in #660

Full Changelog: v0.8.0...v0.8.1

Clusterpedia v0.7.2

What's Changed

• [release/0.7] fix the version order of 'discovery.k8s.io' by @Iceber in #661

Full Changelog: v0.7.1...v0.7.2

Clusterpedia v0.7.2-rc.0

What's Changed

• [release/0.7] fix the version order of 'discovery.k8s.io' by @Iceber in #661

Full Changelog: v0.7.1...v0.7.2-rc.0

Clusterpedia v0.8.0

Notable Changes since v0.7.0

APIService

• add clusterpedia openapi (@wawa0210 @huiwq1990, #579, #588)

The kube apiserver no longer prints the following error log on a recurring basis:

E1225 07:23:43.660066       1 controller.go:116] loading OpenAPI spec for "v1beta1.clusterpedia.io" failed with: OpenAPI spec does not exist
I1225 07:23:43.660079       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.clusterpedia.io: Rate Limited Requeue.

• add OTEL trace config for clusterpedia-apiserver (@KubeKyrie, #604)

Add the --tracing-config-file flag to allow configuration of Otel trace, example configuration

apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
# default
# endpoint: localhost:4317
samplingRatePerMillion: 1000000

Detailed configuration can be found in TracingConfiguration

ClusterSynchrManager

• handle each of the pages in the resource list stage (@Iceber, #591)

This feature will reduce the maximum memory footprint for resource synchronisation.

It is controlled by two feature gates:

• StreamHandlePaginatedListForResourceSync: stream handling of paged resources, resources within a pager will be processed as soon as possible instead of waiting until all resources are pulled before calling the ResourceHandler.
• ForcePaginatedListForResourceSync: force paging, Reflector will sometimes use APIServer's cache, even if paging is specified APIServer will return all resources for performance, then it will skip Reflector's streaming memory optimization.

ForcePaginatedListForResourceSync may put extra pressure on the imported cluster APIServer, please be careful to enable it.

clustersynchro-manager add --page-size flags, you can control the page size of the List, the smaller the page size, the smaller the memory consumption, but may also generate more requests to APIServer.

In the future we may be able to configure these in PediaCluster.

---

• add sharding feature for ClusterSynchroManager (@jxustc, #609)
  In order to avoid a single resource synchroniser taking up a lot of memory, it is now possible to synchronise the PediaCluster in shards.

In two steps:

1. Start a new ClusterSynchroManager Deploymnet or Pod and set the sharding name with --sharding-name flag.
2. Configure the spec.shardingName field of PediaCluster

PediaCluster will only be synchronised by ClusterSynchroManagers with the same sharding name.

---

• add a feature gate IgnoreSyncLease to stop sync leases.coordination.k8s.io (@27149chen, #615, #622)

The leases resource usually needs to be skipped when using resource wildcards or synchronising full resources, and this feature gating ensures that the leases resource is globally skipped without modifying PediaCluster.

In the future, it is more recommended to use #617 to exclude certain resource types.

Multi-Cluster Resource State Metrics

• Support resource state metrics like kube-state-metrics (@Iceber, #539)
• Support for exposing single cluster metrics (@Iceber, #553)
• add kube state metrics namespace command (@scydas, #566)

The cluster synchro has the same source data as kube-state-metrics available to it at sync time, so clusterpedia can easily support functionality similar to kube-state-metrics

# HELP kube_deployment_created [STABLE] Unix creation timestamp
# TYPE kube_deployment_created gauge
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-apiserver"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-clustersynchro-manager"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-controller-manager"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="kube-system",deployment="coredns"} 1.673603507e+09
# HELP kube_deployment_status_replicas [STABLE] The number of replicas per deployment.
# TYPE kube_deployment_status_replicas gauge
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-apiserver"} 1
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-clustersynchro-manager"} 0
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-controller-manager"} 1
kube_deployment_status_replicas{cluster="test-14",namespace="kube-system",deployment="coredns"} 2

Added --enable-kube-state-metrics and a set of related flags to clustersynchro-manager to control multi-cluster kube-state-metrics capability.

$ ./bin/clustersynchro-manager -h
Metrics server flags:
    ...
Kube state metrics flags:
    ...

Storage

• convert deleted objects according to the ConvertDeletedObject method of the storage layer (@huiwq1990 @Iceber, #592, #625)

InternalStorage

• make DB bad connection error message more specific (@KubeKyrie, #598)

Returns a more readable error message(storage error: database connection error: bad connection) in the case of driver.DBError

• Added new parameters for parameterized query (@nekomeowww, #597)

Existing url query - whereSQL needs to splice the value directly in sql, which can easily lead to sql injection problems.

We added three url queries to support parameterized query，give users the ability to prevent sql injection.

url query	desc	example
whereSQLStatement	Splice the sql statement after WHERE,Use ? to represent the value.	whereSQLStatement=(cluster=?) AND (namespace=(?))
whereSQLParam	Multiple whereSQLParam can be set, each corresponding to a question mark.	whereSQLParam=docker-desktop&whereSQLParam=default
whereSQLJSONParams	whereSQLParam has the obvious drawback that it can't pass complex structures and special types such as arrays, etc., so you can use whereSQLJSONParam to do so.	whereSQLStatement=(cluster=?) AND (namespace IN (?))&whereSQLJSONParams=WyJkb2NrZXItZGVza3RvcCIsWyJkZWZhdWx0Iiwia3ViZS1zeXN0ZW0iXV0K

whereSQLJSONParam is a base64 of an array:

$  echo WyJkb2NrZXItZGVza3RvcCIsWyJkZWZhdWx0Iiwia3ViZS1zeXN0ZW0iXV0K | base64 -d
["docker-desktop",["default","kube-system"]]

Parameterised SQL queries are enabled by the AllowParameterizedSQLQuery feature gate

BUG FIX

• fix: direct []byte write when updating resources (@nekomeowww, #607)
• update the created_at field when updating resource (@flpanbin, #629)

Others

• Bump golang from 1.19.5 to 1.21.4 (@dependabot @khareyash05, #537, #552, #581, #587, #610)

• Bump alpine from 3.17.3 to 3.18.4 (@dependabot, #549, #554, #573)

• Bump Kubernetes from v1.26.3 to v1.28.4 (@Iceber @wawa0210, #538, #579, #606)

• Use kustomize to install clusterpedia (@khareyash05 @Iceber, #551, #556, #558)

$ kubectl kustomize ./kustomize | kubectl apply -f -

• Use kustomize for e2e testing and remove the helm chart (@khareyash05, #560, #563)

Contributers

Thanks everyone who contributed to this release!

The following users (sort alphabetically) are those who committed much in this release. Thank you!

• @27149chen
• @flpanbin
• @hexiaodai
• @huiwq1990
• @Iceber
• @jxustc
• @khareyash05
• @KubeKyrie
• @mengjiao-liu
• @nekomeowww
• @rokkiter
• @scydas
• @wawa0210
• @wzshiming

Clusterpedia v0.8.0-beta.0

Notable Changes since v0.7.0

APIService

• add clusterpedia openapi (@wawa0210 @huiwq1990, #579, #588)

The kube apiserver no longer prints the following error log on a recurring basis:

E1225 07:23:43.660066       1 controller.go:116] loading OpenAPI spec for "v1beta1.clusterpedia.io" failed with: OpenAPI spec does not exist
I1225 07:23:43.660079       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.clusterpedia.io: Rate Limited Requeue.

• add OTEL trace config for clusterpedia-apiserver (@KubeKyrie, #604)

Add the --tracing-config-file flag to allow configuration of Otel trace, example configuration

apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
# default
# endpoint: localhost:4317
samplingRatePerMillion: 1000000

Detailed configuration can be found in TracingConfiguration

ClusterSynchrManager

• handle each of the pages in the resource list stage (@Iceber, #591)

This feature will reduce the maximum memory footprint for resource synchronisation.

It is controlled by two feature gates:

• StreamHandlePaginatedListForResourceSync: stream handling of paged resources, resources within a pager will be processed as soon as possible instead of waiting until all resources are pulled before calling the ResourceHandler.
• ForcePaginatedListForResourceSync: force paging, Reflector will sometimes use APIServer's cache, even if paging is specified APIServer will return all resources for performance, then it will skip Reflector's streaming memory optimization.

ForcePaginatedListForResourceSync may put extra pressure on the imported cluster APIServer, please be careful to enable it.

clustersynchro-manager add --page-size flags, you can control the page size of the List, the smaller the page size, the smaller the memory consumption, but may also generate more requests to APIServer.

In the future we may be able to configure these in PediaCluster.

---

• add sharding feature for ClusterSynchroManager (@jxustc, #609)
  In order to avoid a single resource synchroniser taking up a lot of memory, it is now possible to synchronise the PediaCluster in shards.

In two steps:

1. Start a new ClusterSynchroManager Deploymnet or Pod and set the sharding name with --sharding-name flag.
2. Configure the spec.shardingName field of PediaCluster

PediaCluster will only be synchronised by ClusterSynchroManagers with the same sharding name.

---

• add a feature gate IgnoreSyncLease to stop sync leases.coordination.k8s.io (@27149chen, #615, #622)

The leases resource usually needs to be skipped when using resource wildcards or synchronising full resources, and this feature gating ensures that the leases resource is globally skipped without modifying PediaCluster.

In the future, it is more recommended to use #617 to exclude certain resource types.

Multi-Cluster Resource State Metrics

• Support resource state metrics like kube-state-metrics (@Iceber, #539)
• Support for exposing single cluster metrics (@Iceber, #553)
• add kube state metrics namespace command (@scydas, #566)

The cluster synchro has the same source data as kube-state-metrics available to it at sync time, so clusterpedia can easily support functionality similar to kube-state-metrics

# HELP kube_deployment_created [STABLE] Unix creation timestamp
# TYPE kube_deployment_created gauge
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-apiserver"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-clustersynchro-manager"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-controller-manager"} 1.676557618e+09
kube_deployment_created{cluster="test-14",namespace="kube-system",deployment="coredns"} 1.673603507e+09
# HELP kube_deployment_status_replicas [STABLE] The number of replicas per deployment.
# TYPE kube_deployment_status_replicas gauge
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-apiserver"} 1
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-clustersynchro-manager"} 0
kube_deployment_status_replicas{cluster="test-14",namespace="clusterpedia-system",deployment="clusterpedia-controller-manager"} 1
kube_deployment_status_replicas{cluster="test-14",namespace="kube-system",deployment="coredns"} 2

Added --enable-kube-state-metrics and a set of related commands, flags, to clustersynchro-manager to control multi-cluster kube-state-metrics capability.

$ ./bin/clustersynchro-manager -h
Metrics server flags:

      --metrics-disable-gzip-encoding
                Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
      --metrics-host string
                Host to expose clustersynchro manager metrics on. (default "::")
      --metrics-port int
                Port to expose clustersynchro manager metrics on. (default 8081)
      --metrics-tls-config string
                Path to the TLS configuration file of metrics

Kube state metrics flags:

      --enable-kube-state-metrics
                Enabled kube state metrics
      --kube-state-metrics-host string
                Host to expose kube state metrics on. (default "::")
      --kube-state-metrics-metric-allowlist string
                Comma-separated list of metrics to be exposed. This list comprises of exact metric names
                and/or regex patterns. The allowlist and denylist are mutually exclusive.
      --kube-state-metrics-metric-denylist string
                Comma-separated list of metrics not to be enabled. This list comprises of exact metric
                names and/or regex patterns. The allowlist and denylist are mutually exclusive.
      --kube-state-metrics-metric-opt-in-list string
                Comma-separated list of metrics which are opt-in and not enabled by default. This is in
                addition to the metric allow- and denylists
      --kube-state-metrics-namespaces string
                Comma-separated list of namespaces to be enabled. Defaults to ""
      --kube-state-metrics-namespaces-denylist string
                Comma-separated list of namespaces not to be enabled. If namespaces and
                namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist
                will be used.
      --kube-state-metrics-port int
                Port to expose kube state metrics on. (default 8080)
      --kube-state-metrics-resources string
                Comma-separated list of Resources to be enabled. Supported resources:
                "cronjobs,daemonsets,deployments,ingressclasses,ingresses,jobs,namespaces,nodes,pods,replicasets,secrets,services,statefulsets" (default "deployments,pods,services")

Storage

• convert deleted objects according to the ConvertDeletedObject method of the storage layer (@huiwq1990 @Iceber, #592, #625)

InternalStorage

• make DB bad connection error message more specific (@KubeKyrie, #598)

Returns a more readable error message(storage error: database connection error: bad connection) in the case of driver.DBError

• Added new parameters for parameterized query (@nekomeowww, #597)

Existing url query - whereSQL needs to splice the value directly in sql, which can easily lead to sql injection problems.

We added three url queries to support parameterized query，give users the ability to prevent sql injection.

url query	desc	example
whereSQLStatement	Splice the sql statement after WHERE,Use ? to represent the value.	whereSQLStatement=(cluster=?) AND (namespace=(?))
whereSQLParam	Multiple whereSQLParam can be set, each corresponding to a question mark.	whereSQLParam=docker-desktop&whereSQLParam=default
whereSQLJSONParam	whereSQLParam has the obvious drawback that it can't pass complex structures and special types such as arrays, etc., so you can use whereSQLJSONParam to do so.	whereSQLStatement=(cluster=?) AND (namespace IN (?))&whereSQLJSONParams=WyJkb2NrZXItZGVza3RvcCIsWyJkZWZhdWx0Iiwia3ViZS1zeXN0ZW0iXV0K

whereSQLJSONParam is a base64 of an array:

$  echo WyJkb2NrZXItZGVza3RvcCIsWyJkZWZhdWx0Iiwia3ViZS1zeXN0ZW0iXV0K | base64 -d
["docker-desktop",["default","kube-system"]]

Parameterised SQL queries are enabled by the AllowParameterizedSQLQuery feature gate

BUG FIX

• fix: direct []byte write when updating resources (@nekomeowww, #607)
• update the created_at field when updating resource (@flpanbin, #629)

Others

• Bump golang fr...