feature (#916): Observability (#1398)

* Create TelemetryMiddleware

* Refactor TelemetryMiddleware

* Optimization TelemetryMiddleware

* Refactor TelemetryMiddleware

* Change msg type in __call__

* Continue process span while publish

* Change nesting and add create span

* refactor: mv useless types to TYPE_CHECKING

* refactor: add TelemetrySettingsProvider protocol & NATS impl

* chore: update distributions

* Fixed a typo in PR template

* Fixed a typo in ABCBroker: _broker_dependecies to _broker_dependencies

* Add TelemetryTestcase

* Fix invalid ruff formating

* Add telemetry tests, fix kind error

* Add metrics test, fix invalid metric error

* More specifics in the metrics test

* Changed getting a destination to methods

* Add Rabbit impl

* Ruff format

* Remove delivery_tag from publish

* Refactor extracting attributes

* Ruff format

* Add AIOKafka impl

* Add Confluent impl

* Rename test classes

* Add Redis impl

* Refactor settings providers import

* fix: correct TelemetryMiddlewares batch consuming

* Fix spans and metrics ordering in tests, switch Confluent attributes methods

* Add MESSAGING_BATCH_MESSAGE_COUNT attribute

* Add TestConsume and TestPublish to telemetry tests

* refactor: create independant TelemetryMiddleware for each broker

* tests: use broker factory instead of fixtures in Consume and Publish test cases

* tests: refactor RPC and TestClient testcases

* tests: refactor RMQ tests

* tests: refactor Kafka tests

* tests: NATS subscriber tests with patching

* tests: RMQ subscriber tests with patching

* tests: refactor Redis tests

* tests: refactor Confluent tests

* chore: run CI

* tests: fix Confluent TestClient tests

* tests: fix py3.8 compatibility

* lint: fix some mypy

* chore: fix CI

* tests: fix partition tests

* tests: fix partition real broker case

* Change supported metrics

* Separation of metrics for systems with and without batches

* Fix _subscriber_setup_extra merge conflict

* Refactor and update metrics tests, fix metrics

* Refactor metrics, need batches count in publish

* feat: count Redis consuming batch size

* lint: fix mypy

* Add batch tests, metrics ready

* chore: solve main conflicts

* fix: correct confluent batch body serialization

* Fix opentelemetry redis batch test

* lint: correct StreamRoute call annotation

* Add docs for OpenTelemetry

* docs: add otel page links style

* chore: bump version

* Refactor docs

* docs: fix styles

---------

Co-authored-by: Nikita Pastukhov <[email protected]>
Co-authored-by: Pastukhov Nikita <[email protected]>
Co-authored-by: treaditup <[email protected]>

Author: draincoder

.github/PULL_REQUEST_TEMPLATE.md

@@ -22,5 +22,5 @@ Please delete options that are not relevant.
 - [ ] My changes do not generate any new warnings
 - [ ] I have added tests to validate the effectiveness of my fix or the functionality of my new feature
 - [ ] Both new and existing unit tests pass successfully on my local environment by running `scripts/test-cov.sh`
-- [ ] I have ensured that static analysis tests are passing by running `scripts/static-anaylysis.sh`
+- [ ] I have ensured that static analysis tests are passing by running `scripts/static-analysis.sh`
 - [ ] I have included code examples to illustrate the modifications

.github/workflows/test.yaml

@@ -84,7 +84,7 @@ jobs:
           key: ${{ runner.os }}-python-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}-test-v03
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,confluent,nats,redis,testing]
+        run: pip install .[optionals,testing]
       - name: Install Pydantic v1
         if: matrix.pydantic-version == 'pydantic-v1'
         run: pip install "pydantic>=1.10.0,<2.0.0"
@@ -117,7 +117,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing] orjson
+        run: pip install .[optionals,testing] orjson
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m"(slow and (not nats and not kafka and not confluent and not rabbit and not redis)) or (not nats and not kafka and not confluent and not rabbit and not redis)"
@@ -144,7 +144,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,confluent,nats,redis,testing]
+        run: pip install .[optionals,testing]
       - name: Test
         run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not confluent and not rabbit and not redis)) or (not nats and not kafka and not confluent and not rabbit and not redis)"
 
@@ -161,7 +161,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,confluent,nats,redis,testing]
+        run: pip install .[optionals,testing]
       - name: Test
         run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not confluent and not rabbit and not redis)) or (not nats and not kafka and not confluent and not rabbit and not redis)"
 
@@ -194,7 +194,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing]
+        run: pip install .[optionals,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and kafka) or kafka"
@@ -254,7 +254,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing]
+        run: pip install .[optionals,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and confluent) or confluent"
@@ -303,7 +303,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing]
+        run: pip install .[optionals,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and rabbit) or rabbit"
@@ -352,7 +352,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing]
+        run: pip install .[optionals,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and nats) or nats"
@@ -401,7 +401,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,confluent,rabbit,redis,testing]
+        run: pip install .[optionals,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and redis) or redis"

docs/docs/SUMMARY.md

@@ -41,6 +41,7 @@ search:
         - [FastAPI Plugin](getting-started/integrations/fastapi/index.md)
         - [Django](getting-started/integrations/django/index.md)
     - [CLI commands](getting-started/cli/index.md)
+    - [OpenTelemetry](getting-started/opentelemetry/index.md)
     - [Logging](getting-started/logging.md)
     - [Config Management](getting-started/config/index.md)
     - [Task Scheduling](scheduling.md)
@@ -439,6 +440,15 @@ search:
                     - [ConsumerProtocol](api/faststream/confluent/message/ConsumerProtocol.md)
                     - [FakeConsumer](api/faststream/confluent/message/FakeConsumer.md)
                     - [KafkaMessage](api/faststream/confluent/message/KafkaMessage.md)
+                - opentelemetry
+                    - [KafkaTelemetryMiddleware](api/faststream/confluent/opentelemetry/KafkaTelemetryMiddleware.md)
+                    - middleware
+                        - [KafkaTelemetryMiddleware](api/faststream/confluent/opentelemetry/middleware/KafkaTelemetryMiddleware.md)
+                    - provider
+                        - [BaseConfluentTelemetrySettingsProvider](api/faststream/confluent/opentelemetry/provider/BaseConfluentTelemetrySettingsProvider.md)
+                        - [BatchConfluentTelemetrySettingsProvider](api/faststream/confluent/opentelemetry/provider/BatchConfluentTelemetrySettingsProvider.md)
+                        - [ConfluentTelemetrySettingsProvider](api/faststream/confluent/opentelemetry/provider/ConfluentTelemetrySettingsProvider.md)
+                        - [telemetry_attributes_provider_factory](api/faststream/confluent/opentelemetry/provider/telemetry_attributes_provider_factory.md)
                 - parser
                     - [AsyncConfluentParser](api/faststream/confluent/parser/AsyncConfluentParser.md)
                 - publisher
@@ -514,6 +524,15 @@ search:
                     - [ConsumerProtocol](api/faststream/kafka/message/ConsumerProtocol.md)
                     - [FakeConsumer](api/faststream/kafka/message/FakeConsumer.md)
                     - [KafkaMessage](api/faststream/kafka/message/KafkaMessage.md)
+                - opentelemetry
+                    - [KafkaTelemetryMiddleware](api/faststream/kafka/opentelemetry/KafkaTelemetryMiddleware.md)
+                    - middleware
+                        - [KafkaTelemetryMiddleware](api/faststream/kafka/opentelemetry/middleware/KafkaTelemetryMiddleware.md)
+                    - provider
+                        - [BaseKafkaTelemetrySettingsProvider](api/faststream/kafka/opentelemetry/provider/BaseKafkaTelemetrySettingsProvider.md)
+                        - [BatchKafkaTelemetrySettingsProvider](api/faststream/kafka/opentelemetry/provider/BatchKafkaTelemetrySettingsProvider.md)
+                        - [KafkaTelemetrySettingsProvider](api/faststream/kafka/opentelemetry/provider/KafkaTelemetrySettingsProvider.md)
+                        - [telemetry_attributes_provider_factory](api/faststream/kafka/opentelemetry/provider/telemetry_attributes_provider_factory.md)
                 - parser
                     - [AioKafkaParser](api/faststream/kafka/parser/AioKafkaParser.md)
                 - publisher
@@ -596,6 +615,15 @@ search:
                 - message
                     - [NatsBatchMessage](api/faststream/nats/message/NatsBatchMessage.md)
                     - [NatsMessage](api/faststream/nats/message/NatsMessage.md)
+                - opentelemetry
+                    - [NatsTelemetryMiddleware](api/faststream/nats/opentelemetry/NatsTelemetryMiddleware.md)
+                    - middleware
+                        - [NatsTelemetryMiddleware](api/faststream/nats/opentelemetry/middleware/NatsTelemetryMiddleware.md)
+                    - provider
+                        - [BaseNatsTelemetrySettingsProvider](api/faststream/nats/opentelemetry/provider/BaseNatsTelemetrySettingsProvider.md)
+                        - [NatsBatchTelemetrySettingsProvider](api/faststream/nats/opentelemetry/provider/NatsBatchTelemetrySettingsProvider.md)
+                        - [NatsTelemetrySettingsProvider](api/faststream/nats/opentelemetry/provider/NatsTelemetrySettingsProvider.md)
+                        - [telemetry_attributes_provider_factory](api/faststream/nats/opentelemetry/provider/telemetry_attributes_provider_factory.md)
                 - parser
                     - [BatchParser](api/faststream/nats/parser/BatchParser.md)
                     - [JsParser](api/faststream/nats/parser/JsParser.md)
@@ -638,6 +666,16 @@ search:
                     - [PatchedMessage](api/faststream/nats/testing/PatchedMessage.md)
                     - [TestNatsBroker](api/faststream/nats/testing/TestNatsBroker.md)
                     - [build_message](api/faststream/nats/testing/build_message.md)
+            - opentelemetry
+                - [TelemetryMiddleware](api/faststream/opentelemetry/TelemetryMiddleware.md)
+                - [TelemetrySettingsProvider](api/faststream/opentelemetry/TelemetrySettingsProvider.md)
+                - consts
+                    - [MessageAction](api/faststream/opentelemetry/consts/MessageAction.md)
+                - middleware
+                    - [BaseTelemetryMiddleware](api/faststream/opentelemetry/middleware/BaseTelemetryMiddleware.md)
+                    - [TelemetryMiddleware](api/faststream/opentelemetry/middleware/TelemetryMiddleware.md)
+                - provider
+                    - [TelemetrySettingsProvider](api/faststream/opentelemetry/provider/TelemetrySettingsProvider.md)
             - rabbit
                 - [ExchangeType](api/faststream/rabbit/ExchangeType.md)
                 - [RabbitBroker](api/faststream/rabbit/RabbitBroker.md)
@@ -664,6 +702,12 @@ search:
                         - [RabbitRouter](api/faststream/rabbit/fastapi/router/RabbitRouter.md)
                 - message
                     - [RabbitMessage](api/faststream/rabbit/message/RabbitMessage.md)
+                - opentelemetry
+                    - [RabbitTelemetryMiddleware](api/faststream/rabbit/opentelemetry/RabbitTelemetryMiddleware.md)
+                    - middleware
+                        - [RabbitTelemetryMiddleware](api/faststream/rabbit/opentelemetry/middleware/RabbitTelemetryMiddleware.md)
+                    - provider
+                        - [RabbitTelemetrySettingsProvider](api/faststream/rabbit/opentelemetry/provider/RabbitTelemetrySettingsProvider.md)
                 - parser
                     - [AioPikaParser](api/faststream/rabbit/parser/AioPikaParser.md)
                 - publisher
@@ -748,6 +792,12 @@ search:
                     - [StreamMessage](api/faststream/redis/message/StreamMessage.md)
                     - [UnifyRedisDict](api/faststream/redis/message/UnifyRedisDict.md)
                     - [UnifyRedisMessage](api/faststream/redis/message/UnifyRedisMessage.md)
+                - opentelemetry
+                    - [RedisTelemetryMiddleware](api/faststream/redis/opentelemetry/RedisTelemetryMiddleware.md)
+                    - middleware
+                        - [RedisTelemetryMiddleware](api/faststream/redis/opentelemetry/middleware/RedisTelemetryMiddleware.md)
+                    - provider
+                        - [RedisTelemetrySettingsProvider](api/faststream/redis/opentelemetry/provider/RedisTelemetrySettingsProvider.md)
                 - parser
                     - [RawMessage](api/faststream/redis/parser/RawMessage.md)
                     - [RedisBatchListParser](api/faststream/redis/parser/RedisBatchListParser.md)

docs/docs/assets/img/distributed-trace.png

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.KafkaTelemetryMiddleware

docs/docs/assets/img/simple-trace.png

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.middleware.KafkaTelemetryMiddleware

docs/docs/en/api/faststream/confluent/opentelemetry/KafkaTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.provider.BaseConfluentTelemetrySettingsProvider

docs/docs/en/api/faststream/confluent/opentelemetry/middleware/KafkaTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.provider.BatchConfluentTelemetrySettingsProvider

docs/docs/en/api/faststream/confluent/opentelemetry/provider/BaseConfluentTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.provider.ConfluentTelemetrySettingsProvider

docs/docs/en/api/faststream/confluent/opentelemetry/provider/BatchConfluentTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.confluent.opentelemetry.provider.telemetry_attributes_provider_factory

docs/docs/en/api/faststream/confluent/opentelemetry/provider/ConfluentTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.KafkaTelemetryMiddleware

docs/docs/en/api/faststream/confluent/opentelemetry/provider/telemetry_attributes_provider_factory.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.middleware.KafkaTelemetryMiddleware

docs/docs/en/api/faststream/kafka/opentelemetry/KafkaTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.provider.BaseKafkaTelemetrySettingsProvider

docs/docs/en/api/faststream/kafka/opentelemetry/middleware/KafkaTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.provider.BatchKafkaTelemetrySettingsProvider

docs/docs/en/api/faststream/kafka/opentelemetry/provider/BaseKafkaTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.provider.KafkaTelemetrySettingsProvider

docs/docs/en/api/faststream/kafka/opentelemetry/provider/BatchKafkaTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.kafka.opentelemetry.provider.telemetry_attributes_provider_factory

docs/docs/en/api/faststream/kafka/opentelemetry/provider/KafkaTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.nats.opentelemetry.NatsTelemetryMiddleware

docs/docs/en/api/faststream/kafka/opentelemetry/provider/telemetry_attributes_provider_factory.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.nats.opentelemetry.middleware.NatsTelemetryMiddleware

docs/docs/en/api/faststream/nats/opentelemetry/NatsTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.nats.opentelemetry.provider.BaseNatsTelemetrySettingsProvider

docs/docs/en/api/faststream/nats/opentelemetry/middleware/NatsTelemetryMiddleware.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.nats.opentelemetry.provider.NatsBatchTelemetrySettingsProvider

docs/docs/en/api/faststream/nats/opentelemetry/provider/BaseNatsTelemetrySettingsProvider.md

@@ -0,0 +1,11 @@
+---
+# 0.5 - API
+# 2 - Release
+# 3 - Contributing
+# 5 - Template Page
+# 10 - Default
+search:
+  boost: 0.5
+---
+
+::: faststream.nats.opentelemetry.provider.NatsTelemetrySettingsProvider