Skip to content

Plugins Design Refactor #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from
Open

Plugins Design Refactor #26

wants to merge 13 commits into from

Conversation

cccs-mdr
Copy link
Collaborator

@cccs-mdr cccs-mdr commented Jun 5, 2025

No description provided.

Copy link

github-actions bot commented Jun 10, 2025

Static Badge

Howler API - Coverage Results

Static Badge Static Badge

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • api/howler/actions/init.py (100%)
  • api/howler/api/v1/borealis.py (0.0%): Missing lines 13,28-29
  • api/howler/app.py (42.9%): Missing lines 148-149,152,156
  • api/howler/datastore/howler_store.py (55.6%): Missing lines 40-43
  • api/howler/odm/helper.py (75.0%): Missing lines 290
  • api/howler/plugins/init.py (58.8%): Missing lines 16-17,19-23
  • api/howler/plugins/config.py (74.6%): Missing lines 68-70,72,75-77,82-85,87-90,92
  • api/howler/services/config_service.py (100%)
  • api/howler/services/notebook_service.py (100%)

Summary

  • Total: 135 lines
  • Missing: 35 lines
  • Coverage: 74%
api/howler/api/v1/borealis.py

Lines 9-17

   9 from howler.common.exceptions import AuthenticationException
  10 from howler.common.logging import get_logger
  11 from howler.common.swagger import generate_swagger_docs
  12 from howler.config import cache, config
! 13 from howler.plugins import get_plugins
  14 from howler.security import api_login
  15 
  16 SUB_API = "borealis"
  17 borealis_api = make_subapi_blueprint(SUB_API, api_version=1)

Lines 24-33

  24 def get_token(access_token: str) -> str:
  25     """Get a borealis token based on the current howler token"""
  26     get_borealis_token: Optional[Callable[[str], str]] = None
  27 
! 28     for plugin in get_plugins():
! 29         if get_borealis_token := plugin.modules.token_functions.get("borealis", None):
  30             break
  31         else:
  32             logger.info("Plugin %s does not modify the borealis access token.")
api/howler/app.py

Lines 144-160

  144         logger.debug("Enabled Borealis Integration")
  145         app.register_blueprint(borealis_api)
  146 
  147     for plugin in get_plugins():
! 148         if not plugin.modules.routes:
! 149             continue
  150 
  151         try:
! 152             for route in cast(list[Blueprint], plugin.modules.routes):
  153                 logger.info("Enabling additional endpoint: %s", route.url_prefix)
  154                 app.register_blueprint(route)
  155         except ImportError:
! 156             logger.info("Plugin %s does not export additional endpoints.", plugin.name)
  157 
  158 
  159 else:
  160     logger.info("Disabled REST API")
api/howler/datastore/howler_store.py

Lines 36-47

  36     def __init__(self, datastore_object: "ESStore"):
  37         self.ds = datastore_object
  38 
  39         for plugin in get_plugins():
! 40             for _index, _odm in INDEXES:
! 41                 if _odm is not None:
! 42                     if modify_odm := plugin.modules.odm.modify_odm.get(_index):
! 43                         modify_odm(_odm)
  44 
  45         for _index, _odm in INDEXES:
  46             self.ds.register(_index, _odm)
api/howler/odm/helper.py

Lines 286-294

  286 
  287     new_keys: list[str] = []
  288     for plugin in get_plugins():
  289         if generate := plugin.modules.odm.generation.get("hit", None):
! 290             _new_keys, hit = generate(hit)
  291             new_keys += _new_keys
  292 
  293     if len(new_keys) > 0:
  294         logger.debug("%s new top-level fields configured")
api/howler/plugins/__init__.py

Lines 12-26

  12 
  13 def get_plugins() -> list[BasePluginConfig]:
  14     "Get a set of plugin configurations based on the howler settings."
  15     for plugin in _config.core.plugins:
! 16         if plugin in PLUGINS:
! 17             continue
  18 
! 19         try:
! 20             PLUGINS[plugin] = importlib.import_module(f"{plugin}.config").config
! 21         except (ImportError, ModuleNotFoundError):
! 22             logger.exception("Exception when loading plugin")
! 23             PLUGINS[plugin] = None
  24 
  25     return [plugin for plugin in PLUGINS.values() if plugin]
api/howler/plugins/config.py

Lines 64-96

  64 
  65             data["modules"]["routes"] = new_routes
  66 
  67         if "operations" in data["modules"] and isinstance(data["modules"]["operations"], list):
! 68             new_operations: list[str] = []
! 69             for operation in data["modules"]["operations"]:
! 70                 new_operations.append(f"{plugin_name}.actions.{operation}" if "." not in operation else operation)
  71 
! 72             data["modules"]["operations"] = new_operations
  73 
  74         if "token_functions" in data["modules"] and isinstance(data["modules"]["token_functions"], dict):
! 75             for application, value in data["modules"]["token_functions"].items():
! 76                 if value == True:  ## noqa: E712
! 77                     data["modules"]["token_functions"][application] = f"{plugin_name}.token.{value}:get_token"
  78 
  79         if "odm" not in data["modules"] or not isinstance(data["modules"]["odm"], dict):
  80             return data
  81 
! 82         if "modify_odm" in data["modules"]["odm"] and isinstance(data["modules"]["odm"]["modify_odm"], dict):
! 83             for odm_name, value in data["modules"]["odm"]["modify_odm"].items():
! 84                 if data["modules"]["odm"]["modify_odm"][odm_name] == True:  ## noqa: E712
! 85                     data["modules"]["odm"]["modify_odm"][odm_name] = f"{plugin_name}.odm.{odm_name}:modify_odm"
  86 
! 87         if "generation" in data["modules"]["odm"] and isinstance(data["modules"]["odm"]["generation"], dict):
! 88             for odm_name, value in data["modules"]["odm"]["generation"].items():
! 89                 if data["modules"]["odm"]["generation"][odm_name] == True:  ## noqa: E712
! 90                     data["modules"]["odm"]["generation"][odm_name] = f"{plugin_name}.odm.{odm_name}:generate"
  91 
! 92         return data
  93 
  94     @classmethod
  95     def settings_customise_sources(
  96         cls,  ## noqa: ANN102

Full Coverage Report

Expand
Name                                            Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------
howler/__init__.py                                  0      0      0      0   100%
howler/actions/__init__.py                         56      1     26      2    96%
howler/actions/add_label.py                        28      3      6      1    88%
howler/actions/add_to_bundle.py                    41      3     10      2    90%
howler/actions/change_field.py                     18      9      2      0    45%
howler/actions/demote.py                           33     18      8      0    37%
howler/actions/example_plugin.py                   13      8      2      0    33%
howler/actions/prioritization.py                   22     12      2      0    42%
howler/actions/promote.py                          32     18      8      0    35%
howler/actions/remove_from_bundle.py               38      3     10      4    85%
howler/actions/remove_label.py                     28      3      6      2    85%
howler/actions/transition.py                       57      3     22      3    92%
howler/api/__init__.py                            117     47     22      1    55%
howler/api/base.py                                 36      0     12      0   100%
howler/api/socket.py                               65      9     16      6    81%
howler/api/v1/__init__.py                          38      3     14      2    90%
howler/api/v1/action.py                           153     44     52     10    68%
howler/api/v1/analytic.py                         302    108     98     30    59%
howler/api/v1/auth.py                             159     25     54     15    79%
howler/api/v1/borealis.py                          47     47     12      0     0%
howler/api/v1/configs.py                           12      0      0      0   100%
howler/api/v1/dossier.py                           98     41     16      3    53%
howler/api/v1/help.py                              13      0      0      0   100%
howler/api/v1/hit.py                              459     75    156     50    79%
howler/api/v1/notebook.py                          33      5      6      1    85%
howler/api/v1/overview.py                          77     11     18      7    81%
howler/api/v1/search.py                           236     78     62     22    62%
howler/api/v1/template.py                          86     13     26      9    80%
howler/api/v1/tool.py                             101      7     42      5    90%
howler/api/v1/user.py                             175     56     58     23    61%
howler/api/v1/utils/__init__.py                     0      0      0      0   100%
howler/api/v1/utils/etag.py                        23      0      8      1    97%
howler/api/v1/view.py                             127     28     36     14    74%
howler/app.py                                     132     24     36     13    74%
howler/common/__init__.py                           0      0      0      0   100%
howler/common/classification.py                   564     96    312     58    79%
howler/common/exceptions.py                        73      9      2      0    88%
howler/common/hexdump.py                           11      0      0      0   100%
howler/common/iprange.py                          100     15     30      6    82%
howler/common/loader.py                            86     24     40     11    66%
howler/common/logging/__init__.py                 141     83     64      7    36%
howler/common/logging/audit.py                     34      7      8      4    74%
howler/common/logging/format.py                    14      2      0      0    86%
howler/common/net.py                               48      2     22      2    94%
howler/common/net_static.py                         1      0      0      0   100%
howler/common/random_user.py                       10      0      2      0   100%
howler/common/swagger.py                           47     12     18      4    69%
howler/config.py                                   28      0      0      0   100%
howler/cronjobs/__init__.py                        18      2      2      0    90%
howler/cronjobs/retention.py                       30     14      6      2    50%
howler/cronjobs/rules.py                          116      9     36      5    89%
howler/datastore/__init__.py                        0      0      0      0   100%
howler/datastore/bulk.py                           53     42     26      0    14%
howler/datastore/collection.py                   1052    344    500     90    65%
howler/datastore/constants.py                       7      0      0      0   100%
howler/datastore/exceptions.py                     23      1      0      0    96%
howler/datastore/howler_store.py                   96     38     16      1    54%
howler/datastore/operations.py                     45      5     16      5    84%
howler/datastore/schemas.py                         4      4      0      0     0%
howler/datastore/store.py                         117     24     18      3    77%
howler/datastore/support/__init__.py                0      0      0      0   100%
howler/datastore/support/build.py                 101     26     68     10    70%
howler/datastore/support/schemas.py                 4      0      0      0   100%
howler/datastore/types.py                          10      0      0      0   100%
howler/error.py                                    52     31     12      0    33%
howler/healthz.py                                  15      5      2      0    59%
howler/helper/__init__.py                           0      0      0      0   100%
howler/helper/azure.py                             16      9      4      0    35%
howler/helper/discover.py                          28     10      8      0    56%
howler/helper/hit.py                               63     17     30     12    67%
howler/helper/oauth.py                            139     83     88     15    33%
howler/helper/search.py                            20      1      2      1    91%
howler/helper/workflow.py                          47      3     22      3    91%
howler/odm/__init__.py                              1      0      0      0   100%
howler/odm/base.py                                855    168    372     40    76%
howler/odm/helper.py                              171     24     70      7    80%
howler/odm/howler_enum.py                          13      0      0      0   100%
howler/odm/randomizer.py                          217     45    118     18    79%
howler/plugins/__init__.py                         17      7      4      1    52%
howler/plugins/config.py                           63     16     34      4    61%
howler/remote/__init__.py                           0      0      0      0   100%
howler/remote/datatypes/__init__.py                60     18     16      3    67%
howler/remote/datatypes/counters.py                47      5     16      0    92%
howler/remote/datatypes/events.py                  35      1      6      3    90%
howler/remote/datatypes/hash.py                   113     17     22      7    82%
howler/remote/datatypes/lock.py                    18      0      2      0   100%
howler/remote/datatypes/queues/__init__.py          0      0      0      0   100%
howler/remote/datatypes/queues/comms.py            46     10     14      3    75%
howler/remote/datatypes/queues/multi.py            21      3      8      3    79%
howler/remote/datatypes/queues/named.py            66      8     24      8    82%
howler/remote/datatypes/queues/priority.py        107     20     38     12    75%
howler/remote/datatypes/set.py                     73     12      8      2    80%
howler/remote/datatypes/user_quota_tracker.py      27     10      4      0    55%
howler/security/__init__.py                       130     21     42     10    82%
howler/security/socket.py                          50      9      8      4    78%
howler/security/utils.py                           80     37     36      2    47%
howler/services/__init__.py                         0      0      0      0   100%
howler/services/action_service.py                  29      2     10      2    90%
howler/services/analytic_service.py                45      9     16      2    79%
howler/services/auth_service.py                   115     13     48     13    84%
howler/services/config_service.py                  47      8      8      2    82%
howler/services/dossier_service.py                 68      6     32      2    88%
howler/services/event_service.py                   40     13     18      3    62%
howler/services/hit_service.py                    182     18     68     12    86%
howler/services/jwt_service.py                     71     23     18      5    66%
howler/services/lucene_service.py                 149     15     42      6    89%
howler/services/notebook_service.py                58      3     18      3    92%
howler/services/user_service.py                   138     47     62     21    59%
howler/utils/__init__.py                            0      0      0      0   100%
howler/utils/annotations.py                         0      0      0      0   100%
howler/utils/chunk.py                              14      0      8      3    86%
howler/utils/dict_utils.py                        118     17     90     12    82%
howler/utils/isotime.py                             8      1      2      1    80%
howler/utils/list_utils.py                          7      0      4      0   100%
howler/utils/lucene.py                             33      1     12      1    96%
howler/utils/path.py                               16     16      2      0     0%
howler/utils/socket_utils.py                       22      0     14      3    92%
howler/utils/str_utils.py                         115     21     38      4    80%
howler/utils/uid.py                                20      2      4      2    83%
---------------------------------------------------------------------------------
TOTAL                                            9573   2266   3556    689    72%

Copy link

Static Badge

Howler Evidence Plugin - Coverage Results

Static Badge Static Badge

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • plugins/evidence/evidence/config.py (0.0%): Missing lines 2-3,5-6,8-9,11,13,20,23,32

Summary

  • Total: 11 lines
  • Missing: 11 lines
  • Coverage: 0%
plugins/evidence/evidence/config.py

Lines 1-17

   1 ## mypy: ignore-errors
!  2 import os
!  3 from pathlib import Path
   4 
!  5 from howler.plugins.config import BasePluginConfig
!  6 from pydantic_settings import SettingsConfigDict
   7 
!  8 APP_NAME = os.environ.get("APP_NAME", "howler")
!  9 PLUGIN_NAME = "evidence"
  10 
! 11 root_path = Path("/etc") / APP_NAME.replace("-dev", "").replace("-stg", "")
  12 
! 13 config_locations = [
  14     Path(__file__).parent / "manifest.yml",
  15     root_path / "conf" / f"{PLUGIN_NAME}.yml",
  16     Path(os.environ.get("HWL_CONF_FOLDER", root_path)) / f"{PLUGIN_NAME}.yml",
  17 ]

Lines 16-27

  16     Path(os.environ.get("HWL_CONF_FOLDER", root_path)) / f"{PLUGIN_NAME}.yml",
  17 ]
  18 
  19 
! 20 class EvidenceConfig(BasePluginConfig):
  21     "Evidence Plugin Configuration Model"
  22 
! 23     model_config = SettingsConfigDict(
  24         yaml_file=config_locations,
  25         yaml_file_encoding="utf-8",
  26         strict=True,
  27         env_nested_delimiter="__",

Lines 28-36

  28         env_prefix=f"{PLUGIN_NAME}_",
  29     )
  30 
  31 
! 32 config = EvidenceConfig()
  33 
  34 if __name__ == "__main__":
  35     ## When executed, the config model will print the default values of the configuration
  36     import yaml

Full Coverage Report

Expand
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
evidence/__init__.py                  0      0      0      0   100%
evidence/config.py                   11     11      0      0     0%
evidence/odm/hit.py                  13      2      0      0    85%
evidence/odm/models/evidence.py      66      0      0      0   100%
-------------------------------------------------------------------
TOTAL                                90     13      0      0    86%

Copy link

github-actions bot commented Jun 11, 2025

Static Badge

Howler Sentinel Plugin - Coverage Results

Static Badge Static Badge

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • plugins/sentinel/sentinel/actions/send_to_sentinel.py (100%)
  • plugins/sentinel/sentinel/actions/update_defender_xdr_alert.py (100%)
  • plugins/sentinel/sentinel/config.py (100%)
  • plugins/sentinel/sentinel/mapping/sentinel_incident.py (100%)
  • plugins/sentinel/sentinel/mapping/xdr_alert.py (100%)
  • plugins/sentinel/sentinel/odm/hit.py (100%)
  • plugins/sentinel/sentinel/routes/ingest.py (100%)
  • plugins/sentinel/sentinel/utils/tenant_utils.py (100%)

Summary

  • Total: 63 lines
  • Missing: 0 lines
  • Coverage: 100%

Full Coverage Report

Expand
Name                                            Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------
sentinel/__init__.py                                0      0      0      0   100%
sentinel/actions/azure_emit_hash.py                34     22     10      0    27%
sentinel/actions/send_to_sentinel.py               47     14     12      3    68%
sentinel/actions/update_defender_xdr_alert.py      54     18     14      5    63%
sentinel/config.py                                 27      0      0      0   100%
sentinel/mapping/sentinel_incident.py              88     17     30     11    76%
sentinel/mapping/xdr_alert.py                     112     23     48     18    73%
sentinel/mapping/xdr_alert_evidence.py            157     82     34      1    40%
sentinel/odm/__init__.py                            0      0      0      0   100%
sentinel/odm/hit.py                                 8      0      0      0   100%
sentinel/odm/models/sentinel.py                     5      0      0      0   100%
sentinel/routes/__init__.py                         0      0      0      0   100%
sentinel/routes/ingest.py                         109     34     36     12    64%
sentinel/utils/tenant_utils.py                     25      0      8      0   100%
---------------------------------------------------------------------------------
TOTAL                                             666    210    192     50    63%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant