Skip to content

Commit d7041b3

Browse files
dhabledhable
authored
Merge branch 'v2' into dan/hdx-1619-add-api-test-cases-for-the-sources-and-webhook-routes
2 parents 95fe0cf + 1c54b59 commit d7041b3

File tree

13 files changed

+295
-0
lines changed

13 files changed

+295
-0
lines changed

‎docker/otel-collector/config.yaml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,15 @@ receivers:
2727
processors:
2828
transform:
2929
log_statements:
30+
- context: log
31+
error_mode: ignore
32+
statements:
33+
# JSON parsing: Extends log attributes with the fields from structured log body content, either as an OTEL map or
34+
# as a string containing JSON content.
35+
- set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{.*\\}))")) where IsString(log.body)
36+
- merge_maps(log.attributes, ParseJSON(log.cache["0"]), "upsert") where IsMap(log.cache)
37+
- flatten(log.attributes) where IsMap(log.cache)
38+
- merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body)
3039
- context: log
3140
error_mode: ignore
3241
conditions:

‎smoke-tests/otel-collector/README.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# OpenTelemetry Collector Smoke Tests
2+
3+
This directory contains smoke tests for validating the OpenTelemetry Collector functionality in HyperDX.
4+
5+
## Prerequisites
6+
7+
Before running the tests, ensure you have the following tools installed:
8+
9+
- [Bats](https://github.com/bats-core/bats-core) - Bash Automated Testing System
10+
- [Docker](https://www.docker.com/) and Docker Compose
11+
- [curl](https://curl.se/) - Command line tool for transferring data
12+
- [ClickHouse client](https://clickhouse.com/docs/en/integrations/sql-clients/clickhouse-client) - Command-line client for ClickHouse
13+
14+
## Running the Tests
15+
16+
To run all the tests:
17+
18+
```bash
19+
cd smoke-tests/otel-collector
20+
bats *.bats
21+
```
22+
23+
To run a specific test file:
24+
25+
```bash
26+
bats hdx-1453-auto-parse-json.bats
27+
```
28+
29+
## Test Structure
30+
31+
- `*.bats` - Test files written in Bats
32+
- `data/` - Test data used by the tests
33+
- `test_helpers/` - Utility functions for the tests
34+
- `docker-compose.yaml` - Docker Compose configuration for the test environment
35+
36+
## Debugging
37+
38+
If you need to debug the tests, you can set the `SKIP_CLEANUP` environment variable to prevent the Docker containers from being torn down after the tests complete:
39+
40+
```bash
41+
SKIP_CLEANUP=1 bats hdx-1453-auto-parse-json.bats
42+
```
43+
44+
or
45+
46+
```bash
47+
SKIP_CLEANUP=true bats hdx-1453-auto-parse-json.bats
48+
```
49+
50+
With `SKIP_CLEANUP` enabled, the test containers will remain running after the tests complete, allowing you to inspect logs, connect to the containers, and debug issues.
51+
52+
To manually clean up the containers after debugging:
53+
54+
```bash
55+
docker compose down
56+
```

‎smoke-tests/otel-collector/data/auto-parse/default/assert_query.sql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'default' ORDER BY TimestampTime FORMAT CSV

‎smoke-tests/otel-collector/data/auto-parse/default/expected.snap

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
"{'suite-id':'auto-parse','test-id':'default'}","{}"
2+
"{'suite-id':'auto-parse','test-id':'default'}","{}"
3+
"{'suite-id':'auto-parse','test-id':'default'}","{}"
4+
"{'suite-id':'auto-parse','test-id':'default'}","{}"

‎smoke-tests/otel-collector/data/auto-parse/default/input.json

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"resourceLogs": [
3+
{
4+
"resource": {
5+
"attributes": [
6+
{
7+
"key": "suite-id",
8+
"value": {
9+
"stringValue": "auto-parse"
10+
}
11+
},
12+
{
13+
"key": "test-id",
14+
"value": {
15+
"stringValue": "default"
16+
}
17+
}
18+
]
19+
},
20+
"scopeLogs": [
21+
{
22+
"scope": {},
23+
"logRecords": [
24+
{
25+
"timeUnixNano": "1901999580000000000",
26+
"body": {
27+
"stringValue": "[note] this is very much not JSON even though it starts with an array char"
28+
}
29+
},
30+
{
31+
"timeUnixNano": "1901999580000000001",
32+
"body": {
33+
"stringValue": "{note} this is very much not JSON even though it starts with an object char"
34+
}
35+
},
36+
{
37+
"timeUnixNano": "1901999580000000002",
38+
"body": {
39+
"stringValue": "NOTE: this is very much not JSON"
40+
}
41+
},
42+
{
43+
"timeUnixNano": "1901999580000000003",
44+
"body": {
45+
"stringValue": "this has some {Key {Value { '{' } } invalid JSON in it"
46+
}
47+
}
48+
]
49+
}
50+
]
51+
}
52+
]
53+
}

‎smoke-tests/otel-collector/data/auto-parse/json-string/assert_query.sql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'json-string' ORDER BY TimestampTime FORMAT CSV

‎smoke-tests/otel-collector/data/auto-parse/json-string/expected.snap

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
"{'suite-id':'auto-parse','test-id':'json-string'}","{'attr.intValue':'1','found':'false','message':'this should be parsed into a map'}"
2+
"{'suite-id':'auto-parse','test-id':'json-string'}","{'bodyAttr':'12345','message':'this has an existing user attribute that should be preserved.','userAttr':'true'}"
3+
"{'suite-id':'auto-parse','test-id':'json-string'}","{'found':'true','position':'trailing'}"
4+
"{'suite-id':'auto-parse','test-id':'json-string'}","{'found':'true','position':'leading'}"
5+
"{'suite-id':'auto-parse','test-id':'json-string'}","{'found':'true','position':'wrapped'}"

‎smoke-tests/otel-collector/data/auto-parse/json-string/input.json

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
{
2+
"resourceLogs": [
3+
{
4+
"resource": {
5+
"attributes": [
6+
{
7+
"key": "suite-id",
8+
"value": {
9+
"stringValue": "auto-parse"
10+
}
11+
},
12+
{
13+
"key": "test-id",
14+
"value": {
15+
"stringValue": "json-string"
16+
}
17+
}
18+
]
19+
},
20+
"scopeLogs": [
21+
{
22+
"scope": {},
23+
"logRecords": [
24+
{
25+
"timeUnixNano": "1901999580000000000",
26+
"body": {
27+
"stringValue": "{\"attr\":{\"intValue\": 1},\"found\":false,\"message\":\"this should be parsed into a map\"}"
28+
}
29+
},
30+
{
31+
"timeUnixNano": "1901999580000000001",
32+
"attributes": [
33+
{
34+
"key": "userAttr",
35+
"value": {
36+
"boolValue": true
37+
}
38+
}
39+
],
40+
"body": {
41+
"stringValue": "{\"bodyAttr\":12345,\"message\":\"this has an existing user attribute that should be preserved.\"}"
42+
}
43+
},
44+
{
45+
"timeUnixNano": "1901999580000000002",
46+
"body": {
47+
"stringValue": "should find the trailing JSON object {\"found\":true,\"position\":\"trailing\"}"
48+
}
49+
},
50+
{
51+
"timeUnixNano": "1901999580000000003",
52+
"body": {
53+
"stringValue": "{\"found\":true,\"position\":\"leading\"} should find the leading JSON object "
54+
}
55+
},
56+
{
57+
"timeUnixNano": "1901999580000000004",
58+
"body": {
59+
"stringValue": "should find a wrapped JSON object {\"found\":true,\"position\":\"wrapped\"} between text"
60+
}
61+
}
62+
]
63+
}
64+
]
65+
}
66+
]
67+
}

‎smoke-tests/otel-collector/data/auto-parse/otel-map/assert_query.sql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'otel-map' ORDER BY TimestampTime FORMAT CSV

‎smoke-tests/otel-collector/data/auto-parse/otel-map/expected.snap

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"{'suite-id':'auto-parse','test-id':'otel-map'}","{'account-id':'550e8400-e29b-41d4-a716-446655440000','message':'data sent as OTEL map should also extend the log attributes','user-id':'1234'}"

0 commit comments

Comments
 (0)