Skip to content

Commit 6bdc5e5

Browse files
schmikeidjaglowski
andauthored
[receiver/vcenter] Construct MetricsReceiver Component (#9224)
* add vcenter vSAN collection * checkpoint on getting property collection working * checkpoint before integration test * dual receivers under root receiver pointer * checkpoint before updated mdatagen * use syslog receiver rather than tcplogreceiver * getting more performance counter refinements * remove unneccessary component addition * try to fix go.mod resolution issues * try to fix go.mod resolution issues pt 2 * addlicense * fix go.mod by fixing require directive * add readme for metrics * update readme * fix go.mod referring nonexistent version * add performance manager tests * more tests * add more attributes to virtual machines and host systems * add more attributes to virtual machines and host systems * spike changelog entry * fix go.mod in both places * fix go.mod in configschema * add // import github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vmwarevcenterreceiver to imports * add quotations * add to receiver lifecycle * remove extra go generate direction * fix typo of utilizaiton in metric description * small changes to interval id in performance queries to be more consistent * PR feedback including omitting company name prefix * PR feedback to not fail starting the component on potential network failures * minor grammar correction in vcenter readme * update expected metrics * update host_effective attribute value * remove PerformanceInterval customizability * add to codeowners * fix indentation on merge conflict * fix changelog entry place so its in the new components section * update to be on 0.49.0 of the collector' * add PR number to changelog * regenerate with newer version of mdatagen * move error log if unable to connect on start to receiver.Start() rather than scraper.Start() * fix test cases from last commit * minor update to config with tests * fix metric description * use utc for host vsan collection as well * update comments of public facing methods * return errors on getting clusters to the scraper errors * PR feedback #1 * instantiate new client if client is nil * update all descriptions to have punctuation * three more descs * move ensureReceiver up to once we validated as a config * some more PR feedback * looking into race conditions * run go tidy * fix import order and remove unneccessary mutex * remove mutex from struct * refactor client to responsible for knowing if the vsan endpoints are reachable * fix integration test referencing old var * change metrics.metrics => metrics.settings, update client pr feedback * remove vSAN collection temporarily * remove extra metric attributes for vSAN * remove vsan specific variables * clean up host PerfCounter disk latency metrics and fix some descriptions to better reflect interval * add 20s interval to extended documentation as needed * mdatagen fixes * add integration test metric scrape * fix import order * go up to 0.49.1 * gotidy * add replace directive for semconv * gotidy fixes * fix component not being on 0.50.0 * update to v0.50.1-0.20220429151328-041f39835df7 * use newer mdatagen * remove any logging functionality change && update documentation * fix integration test from flattening of config * fix scraper start not erroring if connection cannot be established * make scrapertest less flaky * format test json * Apply suggestions from code review Co-authored-by: Daniel Jaglowski <[email protected]> * adjust metric definition for vcenter.host.disk.throughput * remove comment and move pm level 2 metrics to appropriate section * try to be respective of datacenters * fix only vCenter server functionality * try building out a mock server for test coverage * make goporto * fix build issues * use latest mdatagen * add newlines to ends of xml recordings * fix integration test * moved around scrapererrors because now the receiver is datacenter dependent * try and do an audit of performance metrics and requests/responses * update testdata with correct units * make tidy * make tidy * update collector version * fix local testing code including modules * remove deprecated use of commonponenterror * pr feedback; add method of collection recording, return poweredOn/poweredOff VMs * remove content.json * fix description change in scraper_test.go * update collector version * bump replaced module; rebuild load tests * fix alibaba version auto localizing Co-authored-by: Daniel Jaglowski <[email protected]>
1 parent 41e4235 commit 6bdc5e5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+55337
-0
lines changed

.github/CODEOWNERS

+1
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,7 @@ receiver/statsdreceiver/ @open-telemetry/collector-c
162162
receiver/syslogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
163163
receiver/tcplogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
164164
receiver/udplogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
165+
receiver/vcenterreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski @schmikei
165166
receiver/wavefrontreceiver/ @open-telemetry/collector-contrib-approvers @pjanotti
166167
receiver/windowseventlogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski @armstrmi
167168
receiver/windowsperfcountersreceiver/ @open-telemetry/collector-contrib-approvers @dashpole

CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,7 @@
128128
### 🚀 New components 🚀
129129

130130
- `iisreceiver`: Add implementation of IIS Metric Receiver (#8832)
131+
- `vcenterreceiver`: Add metrics receiver for new vcenterreceiver component (#9224)
131132
- `sqlserverreceiver`: Add implementation of SQL Server Metric Receiver (#8398)
132133
- `activedirectorydsreceiver`: Add implementation of Active Directory Domain Services metric receiver (#9359)
133134

cmd/configschema/go.mod

+4
Original file line numberDiff line numberDiff line change
@@ -385,6 +385,7 @@ require (
385385
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.52.0 // indirect
386386
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplogreceiver v0.52.0 // indirect
387387
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver v0.52.0 // indirect
388+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.52.0 // indirect
388389
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver v0.52.0 // indirect
389390
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver v0.52.0 // indirect
390391
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowsperfcountersreceiver v0.52.0 // indirect
@@ -456,6 +457,7 @@ require (
456457
github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
457458
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
458459
github.com/vmware/go-vmware-nsxt v0.0.0-20220328155605-f49a14c1ef5f // indirect
460+
github.com/vmware/govmomi v0.28.0 // indirect
459461
github.com/wavefronthq/wavefront-sdk-go v0.9.11 // indirect
460462
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
461463
github.com/xdg-go/scram v1.1.1 // indirect
@@ -840,6 +842,8 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplo
840842

841843
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver => ../../receiver/udplogreceiver
842844

845+
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver => ../../receiver/vcenterreceiver
846+
843847
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver => ../../receiver/wavefrontreceiver
844848

845849
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver => ../../receiver/windowseventlogreceiver

cmd/configschema/go.sum

+3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go.mod

+4
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ require (
132132
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.52.0
133133
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplogreceiver v0.52.0
134134
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver v0.52.0
135+
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.52.0
135136
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver v0.52.0
136137
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver v0.52.0
137138
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowsperfcountersreceiver v0.52.0
@@ -457,6 +458,7 @@ require (
457458
github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
458459
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
459460
github.com/vmware/go-vmware-nsxt v0.0.0-20220328155605-f49a14c1ef5f // indirect
461+
github.com/vmware/govmomi v0.28.0 // indirect
460462
github.com/wavefronthq/wavefront-sdk-go v0.9.11 // indirect
461463
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
462464
github.com/xdg-go/scram v1.1.1 // indirect
@@ -842,6 +844,8 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplo
842844

843845
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver => ./receiver/udplogreceiver
844846

847+
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver => ./receiver/vcenterreceiver
848+
845849
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver => ./receiver/wavefrontreceiver
846850

847851
replace github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver => ./receiver/windowseventlogreceiver

go.sum

+3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

internal/components/components.go

+2
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,7 @@ import (
152152
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver"
153153
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplogreceiver"
154154
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver"
155+
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver"
155156
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver"
156157
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver"
157158
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowsperfcountersreceiver"
@@ -247,6 +248,7 @@ func Components() (component.Factories, error) {
247248
syslogreceiver.NewFactory(),
248249
tcplogreceiver.NewFactory(),
249250
udplogreceiver.NewFactory(),
251+
vcenterreceiver.NewFactory(),
250252
zipkinreceiver.NewFactory(),
251253
}
252254
receivers = append(receivers, extraReceivers()...)

internal/components/receivers_test.go

+3
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,9 @@ func TestDefaultReceivers(t *testing.T) {
300300
return cfg
301301
},
302302
},
303+
{
304+
receiver: "vcenter",
305+
},
303306
}
304307

305308
assert.Len(t, tests, len(rcvrFactories), "All receivers must be added to the lifecycle suite")

receiver/vcenterreceiver/Makefile

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
include ../../Makefile.Common

receiver/vcenterreceiver/README.md

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# vCenter Receiver
2+
3+
This receiver fetches metrics from a vCenter or ESXi host running VMware vSphere APIs.
4+
5+
Supported pipeline types: `metrics`
6+
7+
> :construction: This receiver is in **Alpha**. Configuration fields and metric data model are subject to change.
8+
9+
## Prerequisites
10+
11+
This receiver has been built to support ESXi and vCenter versions:
12+
13+
- 7.5
14+
- 7.0
15+
- 6.7
16+
17+
A “Read Only” user assigned to a vSphere with permissions to the vCenter server, cluster and all subsequent resources being monitored must be specified in order for the receiver to retrieve information about them.
18+
19+
## Configuration
20+
21+
22+
| Parameter | Default | Type | Notes |
23+
| --- | --- | --- | --- |
24+
| endpoint | | String | Endpoint to the vCenter Server or ESXi host that has the sdk path enabled. Required. The expected format is `<protocol>://<hostname>` <br><br> i.e: `https://vcsa.hostname.localnet` |
25+
| username | | String | Required |
26+
| password | | String | Required |
27+
| tls | | TLSClientSetting | Not Required. Will use defaults for [configtls.TLSClientSetting](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md). By default insecure settings are rejected and certificate verification is on. |
28+
| collection_interval | 2m | Duration | This receiver collects metrics on an interval. If the vCenter is fairly large, this value may need to be increased. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h` |
29+
30+
### Example Configuration
31+
32+
```yaml
33+
receivers:
34+
vcenter:
35+
endpoint: http://localhost:15672
36+
username: otelu
37+
password: $VCENTER_PASSWORD
38+
collection_interval: 5m
39+
metrics: []
40+
```
41+
42+
The full list of settings exposed for this receiver are documented [here](./config.go) with detailed sample configurations [here](./testdata/config.yaml). TLS config is documented further under the [opentelemetry collector's configtls package](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md).
43+
44+
## Metrics
45+
46+
Details about the metrics produced by this receiver can be found in [metadata.yaml](./metadata.yaml) with further documentation in [documentation.md](./documentation.md)

receiver/vcenterreceiver/client.go

+154
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
// Copyright The OpenTelemetry Authors
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package vcenterreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver"
16+
17+
import (
18+
"context"
19+
"fmt"
20+
"net/url"
21+
22+
"github.com/vmware/govmomi"
23+
"github.com/vmware/govmomi/find"
24+
"github.com/vmware/govmomi/object"
25+
"github.com/vmware/govmomi/performance"
26+
"github.com/vmware/govmomi/property"
27+
"github.com/vmware/govmomi/vim25"
28+
vt "github.com/vmware/govmomi/vim25/types"
29+
)
30+
31+
// vcenterClient is a client that
32+
type vcenterClient struct {
33+
moClient *govmomi.Client
34+
vimDriver *vim25.Client
35+
finder *find.Finder
36+
pc *property.Collector
37+
pm *performance.Manager
38+
cfg *Config
39+
}
40+
41+
func newVcenterClient(c *Config) *vcenterClient {
42+
return &vcenterClient{
43+
cfg: c,
44+
}
45+
}
46+
47+
// EnsureConnection will establish a connection to the vSphere SDK if not already established
48+
func (vc *vcenterClient) EnsureConnection(ctx context.Context) error {
49+
if vc.moClient != nil {
50+
return nil
51+
}
52+
53+
sdkURL, err := vc.cfg.SDKUrl()
54+
if err != nil {
55+
return err
56+
}
57+
client, err := govmomi.NewClient(ctx, sdkURL, vc.cfg.Insecure)
58+
if err != nil {
59+
return fmt.Errorf("unable to connect to vSphere SDK on listed endpoint: %w", err)
60+
}
61+
tlsCfg, err := vc.cfg.LoadTLSConfig()
62+
if err != nil {
63+
return err
64+
}
65+
client.DefaultTransport().TLSClientConfig = tlsCfg
66+
user := url.UserPassword(vc.cfg.Username, vc.cfg.Password)
67+
err = client.Login(ctx, user)
68+
if err != nil {
69+
return fmt.Errorf("unable to login to vcenter sdk: %w", err)
70+
}
71+
vc.moClient = client
72+
vc.vimDriver = client.Client
73+
vc.pc = property.DefaultCollector(vc.vimDriver)
74+
vc.finder = find.NewFinder(vc.vimDriver)
75+
vc.pm = performance.NewManager(vc.vimDriver)
76+
return nil
77+
}
78+
79+
// Disconnect will logout of the autenticated session
80+
func (vc *vcenterClient) Disconnect(ctx context.Context) error {
81+
if vc.moClient != nil {
82+
return vc.moClient.Logout(ctx)
83+
}
84+
return nil
85+
}
86+
87+
// Clusters returns the clusterComputeResources of the vSphere SDK
88+
func (vc *vcenterClient) Datacenters(ctx context.Context) ([]*object.Datacenter, error) {
89+
datacenters, err := vc.finder.DatacenterList(ctx, "*")
90+
if err != nil {
91+
return []*object.Datacenter{}, fmt.Errorf("unable to get datacenter lists: %w", err)
92+
}
93+
return datacenters, nil
94+
}
95+
96+
// Clusters returns the clusterComputeResources of the vSphere SDK
97+
func (vc *vcenterClient) Clusters(ctx context.Context, datacenter *object.Datacenter) ([]*object.ClusterComputeResource, error) {
98+
vc.finder = vc.finder.SetDatacenter(datacenter)
99+
clusters, err := vc.finder.ClusterComputeResourceList(ctx, "*")
100+
if err != nil {
101+
return []*object.ClusterComputeResource{}, err
102+
}
103+
return clusters, nil
104+
}
105+
106+
// ResourcePools returns the resourcePools in the vSphere SDK
107+
func (vc *vcenterClient) ResourcePools(ctx context.Context) ([]*object.ResourcePool, error) {
108+
rps, err := vc.finder.ResourcePoolList(ctx, "*")
109+
if err != nil {
110+
return nil, fmt.Errorf("unable to retrieve resource pools: %w", err)
111+
}
112+
return rps, err
113+
}
114+
115+
func (vc *vcenterClient) VMs(ctx context.Context) ([]*object.VirtualMachine, error) {
116+
vms, err := vc.finder.VirtualMachineList(ctx, "*")
117+
if err != nil {
118+
return nil, fmt.Errorf("unable to retrieve resource pools: %w", err)
119+
}
120+
return vms, err
121+
}
122+
123+
type perfSampleResult struct {
124+
counters map[string]*vt.PerfCounterInfo
125+
results []performance.EntityMetric
126+
}
127+
128+
func (vc *vcenterClient) performanceQuery(
129+
ctx context.Context,
130+
spec vt.PerfQuerySpec,
131+
names []string,
132+
objs []vt.ManagedObjectReference,
133+
) (*perfSampleResult, error) {
134+
if vc.pm == nil {
135+
return &perfSampleResult{}, nil
136+
}
137+
vc.pm.Sort = true
138+
sample, err := vc.pm.SampleByName(ctx, spec, names, objs)
139+
if err != nil {
140+
return nil, err
141+
}
142+
result, err := vc.pm.ToMetricSeries(ctx, sample)
143+
if err != nil {
144+
return nil, err
145+
}
146+
counterInfoByName, err := vc.pm.CounterInfoByName(ctx)
147+
if err != nil {
148+
return nil, err
149+
}
150+
return &perfSampleResult{
151+
counters: counterInfoByName,
152+
results: result,
153+
}, nil
154+
}
+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Copyright The OpenTelemetry Authors
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package vcenterreceiver // import github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver
16+
17+
import (
18+
"context"
19+
"testing"
20+
21+
"github.com/stretchr/testify/require"
22+
"github.com/vmware/govmomi/find"
23+
"github.com/vmware/govmomi/simulator"
24+
"github.com/vmware/govmomi/vim25"
25+
)
26+
27+
func TestGetClusters(t *testing.T) {
28+
simulator.Test(func(ctx context.Context, c *vim25.Client) {
29+
finder := find.NewFinder(c)
30+
client := vcenterClient{
31+
vimDriver: c,
32+
finder: finder,
33+
}
34+
dc, err := finder.DefaultDatacenter(ctx)
35+
require.NoError(t, err)
36+
clusters, err := client.Clusters(ctx, dc)
37+
require.NoError(t, err)
38+
require.NotEmpty(t, clusters, 0)
39+
})
40+
}
41+
42+
func TestGetResourcePools(t *testing.T) {
43+
simulator.Test(func(ctx context.Context, c *vim25.Client) {
44+
finder := find.NewFinder(c)
45+
client := vcenterClient{
46+
vimDriver: c,
47+
finder: finder,
48+
}
49+
resourcePools, err := client.ResourcePools(ctx)
50+
require.NoError(t, err)
51+
require.NotEmpty(t, resourcePools)
52+
})
53+
}
54+
55+
func TestGetVMs(t *testing.T) {
56+
simulator.Test(func(ctx context.Context, c *vim25.Client) {
57+
finder := find.NewFinder(c)
58+
client := vcenterClient{
59+
vimDriver: c,
60+
finder: finder,
61+
}
62+
vms, err := client.VMs(ctx)
63+
require.NoError(t, err)
64+
require.NotEmpty(t, vms)
65+
})
66+
}

0 commit comments

Comments
 (0)