Skip to content

feat: add DMQ publisher/consumer in signer/aggregator #2599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
Jul 1, 2025

Conversation

jpraynaud
Copy link
Member

@jpraynaud jpraynaud commented Jun 20, 2025

Content

This PR includes the implementation of the signature publisher and consumers based on the Pallas DMQ implementation of the n2c Local Message Submission and n2c Local Message Notification in the signer and aggregator:

  • Added a new mithril-dmq-node internal crate to support the publication and notification of generic messages through the DMQ network
  • Implemented a DMQ publisher in the signer
  • Implemented a DMQ consumer in the aggregator

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Closes #2539 and #2540

@jpraynaud jpraynaud self-assigned this Jun 20, 2025
@jpraynaud jpraynaud requested a review from Copilot June 20, 2025 16:14
Copilot

This comment was marked as outdated.

@jpraynaud jpraynaud added the prototype 🛠️ Prototype/PoC of a feature label Jun 20, 2025
Copy link

github-actions bot commented Jun 20, 2025

Test Results

    3 files  ± 0    105 suites  +3   17m 59s ⏱️ +31s
2 071 tests +18  2 071 ✅ +18  0 💤 ±0  0 ❌ ±0 
4 285 runs  +41  4 285 ✅ +41  0 💤 ±0  0 ❌ ±0 

Results for commit 36f6e67. ± Comparison against base commit 06a9beb.

This pull request removes 2 and adds 20 tests. Note that renamed tests count towards both.
mithril-common ‑ messages::register_signature::tests::http_message::test_current_json_deserialized_into_current_message
mithril-common ‑ messages::register_signature::tests::http_message::test_json_until_open_api_0_1_45_deserialized_into_current_message
mithril-aggregator ‑ services::signature_consumer::dmq::tests::get_signatures_failure
mithril-aggregator ‑ services::signature_consumer::dmq::tests::get_signatures_success
mithril-common ‑ crypto_helper::cardano::kes::signer_fake::tests::fake_kes_signer_returns_signature_batches_in_expected_order
mithril-common ‑ entities::signed_entity_type::tests::bytes_encoding
mithril-common ‑ messages::register_signature::tests::golden_dmq_message::test_current_bijective_bytes_codec
mithril-common ‑ messages::register_signature::tests::golden_dmq_message::test_current_bytes_decoded_into_current_message
mithril-common ‑ messages::register_signature::tests::golden_http_message::test_current_json_deserialized_into_current_message
mithril-common ‑ messages::register_signature::tests::golden_http_message::test_json_until_open_api_0_1_45_deserialized_into_current_message
mithril-dmq ‑ consumer::pallas::tests::pallas_dmq_consumer_client_is_dropped_when_returning_error
mithril-dmq ‑ consumer::pallas::tests::pallas_dmq_consumer_publisher_blocks_when_no_message_available
…

♻️ This comment has been updated with latest results.

@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch 3 times, most recently from 12ea76a to c2f2a24 Compare June 23, 2025 17:18
@jpraynaud jpraynaud requested a review from Copilot June 23, 2025 17:19
Copilot

This comment was marked as outdated.

@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from c2f2a24 to 1e4cf90 Compare June 24, 2025 15:15
@jpraynaud jpraynaud temporarily deployed to testing-preview June 24, 2025 15:26 — with GitHub Actions Inactive
@jpraynaud jpraynaud requested a review from Copilot June 24, 2025 15:28
Copilot

This comment was marked as outdated.

@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from 1e4cf90 to b849730 Compare June 25, 2025 16:47
@jpraynaud jpraynaud temporarily deployed to testing-preview June 25, 2025 16:59 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from b849730 to 2db69f2 Compare June 26, 2025 15:20
@jpraynaud jpraynaud removed the prototype 🛠️ Prototype/PoC of a feature label Jun 26, 2025
@jpraynaud jpraynaud requested a review from Copilot June 26, 2025 15:23
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for publishing and consuming signatures over a Decentralized Message Queue (DMQ) in both the signer and aggregator nodes, wiring the new DMQ-based publishers/consumers through configuration and dependency injection.

  • Introduce DMQ-based SignaturePublisher and SignatureConsumer implementations with full serialization, deserialization, and tests
  • Extend common message definitions (RegisterSignatureMessageDmq) and SignedEntityType with byte-codec support and golden tests
  • Update configuration, DI builders, and runtime wiring to conditionally enable DMQ paths when dmq_node_socket_path is set

Reviewed Changes

Copilot reviewed 49 out of 51 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
mithril-signer/src/services/signature_publisher/dmq.rs New DMQ-based SignaturePublisher implementation
mithril-aggregator/src/services/signature_consumer/dmq.rs New DMQ-based SignatureConsumer implementation
mithril-common/src/messages/register_signature.rs Added RegisterSignatureMessageDmq type and (de)serialization
mithril-signer/src/dependency_injection/builder.rs Wired DMQ publisher into signer DI builder
mithril-aggregator/src/dependency_injection/builder/enablers/misc.rs Wired DMQ consumer into aggregator DI builder
mithril-signer/src/configuration.rs & mithril-aggregator/src/configuration.rs Added dmq_node_socket_path to configuration schemas
Comments suppressed due to low confidence (4)

mithril-signer/src/services/signature_publisher/dmq.rs:15

  • [nitpick] Use the uppercase acronym DMQ instead of Dmq in the doc comment to match the rest of the codebase.
/// Dmq implementation of the [SignaturePublisher] trait.

mithril-aggregator/src/dependency_injection/builder/enablers/misc.rs:81

  • Consider adding unit tests for build_signature_consumer to cover both the DMQ branch (when dmq_node_socket_path is set) and the Noop branch.
            if let Some(dmq_node_socket_path) = self.configuration.dmq_node_socket_path() {

mithril-signer/src/dependency_injection/builder.rs:397

  • Add tests for the signer DI builder to ensure that the DMQ-based SignaturePublisher is instantiated when dmq_node_socket_path is provided and fallback Noop otherwise.
        let kes_signer = match (

mithril-signer/src/dependency_injection/builder.rs:397

  • [nitpick] The DMQ publisher setup block is similar between signer and aggregator builders; consider extracting a shared helper to avoid duplication.
        let kes_signer = match (

@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from 2db69f2 to 74bdcf3 Compare June 26, 2025 15:30
@jpraynaud jpraynaud temporarily deployed to testing-preview June 26, 2025 15:39 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from 74bdcf3 to 0126b02 Compare June 26, 2025 16:02
@jpraynaud jpraynaud marked this pull request as ready for review June 26, 2025 16:03
@jpraynaud jpraynaud temporarily deployed to testing-preview June 26, 2025 16:11 — with GitHub Actions Inactive
jpraynaud and others added 24 commits July 1, 2025 09:50
* mithril-dmq from `0.1.0` to `0.1.1`
* mithril-aggregator from `0.7.69` to `0.7.70`
* mithril-common from `0.6.3` to `0.6.4`
* mithril-signer from `0.2.256` to `0.2.257`
@jpraynaud jpraynaud force-pushed the jpraynaud/2402-publisher-consumer-dmq branch from e61ee88 to 36f6e67 Compare July 1, 2025 07:51
@jpraynaud jpraynaud temporarily deployed to testing-preview July 1, 2025 07:59 — with GitHub Actions Inactive
@jpraynaud jpraynaud merged commit 55ca841 into main Jul 1, 2025
39 of 40 checks passed
@jpraynaud jpraynaud deleted the jpraynaud/2402-publisher-consumer-dmq branch July 1, 2025 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants