Fix issue computing diffs in compliance audit log when writing to security index

[cwperks]

Description

I encountered this issue when attempting to build alerts for changes in the security index.

The compliance audit log can be configured to audit log diffs to selected indices and by default includes the security index. Make sure that config.compliance.enabled and config.compliance.write_log_diffs are set to true and that config.compliance.write_metadata_only is set to false.

Sample audit log config

_meta:
  type: "audit"
  config_version: 2

config:
  # enable/disable audit logging
  enabled: true

  audit:
    # Enable/disable REST API auditing
    enable_rest: true

    # Categories to exclude from REST API auditing
    disabled_rest_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

    # Enable/disable Transport API auditing
    enable_transport: true

    # Categories to exclude from Transport API auditing
    disabled_transport_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

    # Users to be excluded from auditing. Wildcard patterns are supported. Eg:
    # ignore_users: ["test-user", "employee-*"]
    ignore_users:
      - kibanaserver

    # Requests to be excluded from auditing. Wildcard patterns are supported. Eg:
    # ignore_requests: ["indices:data/read/*", "SearchRequest"]
    # ignore_requests: []

    # Log individual operations in a bulk request
    resolve_bulk_requests: false

    # Include the body of the request (if available) for both REST and the transport layer
    log_request_body: false

    # Logs all indices affected by a request. Resolves aliases and wildcards/date patterns
    resolve_indices: true

    # Exclude sensitive headers from being included in the logs. Eg: Authorization
    exclude_sensitive_headers: true

  compliance:
    # enable/disable compliance
    enabled: true

    # Log updates to internal security changes
    internal_config: true

    # Log external config files for the node
    external_config: false

    # Log only metadata of the document for read events
    read_metadata_only: true

    # Map of indexes and fields to monitor for read events. Wildcard patterns are supported for both index names and fields. Eg:
    # read_watched_fields: {
    #   "twitter": ["message"]
    #   "logs-*": ["id", "attr*"]
    # }
    read_watched_fields: {}

    # List of users to ignore for read events. Wildcard patterns are supported. Eg:
    # read_ignore_users: ["test-user", "employee-*"]
    read_ignore_users:
      - kibanaserver

    # Log only metadata of the document for write events
    write_metadata_only: false

    # Log only diffs for document updates
    write_log_diffs: true

    # List of indices to watch for write events. Wildcard patterns are supported
    # write_watched_indices: ["twitter", "logs-*"]
    write_watched_indices: ["movies"]

    # List of users to ignore for write events. Wildcard patterns are supported. Eg:
    # write_ignore_users: ["test-user", "employee-*"]
    write_ignore_users:
      - kibanaserver

Currently, when there are writes to the security index, the ComplianceIndexingOperationListenerImpl errantly thinks that the index is empty in the previous state and the diffs will always include the entirety of the document (ctype) being updated. The reason for that is that the call to getForUpdate is silently failing and returning an empty document. The changes in this PR ensure that we can reliably get the existing state of the document (ctype) being updated.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Bug fix

Issues Resolved

Resolves #5280

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.99%. Comparing base (cac58bc) to head (333d5ad).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
...figuration/SecurityFlsDlsIndexSearcherWrapper.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5279      +/-   ##
==========================================
- Coverage   72.04%   71.99%   -0.05%     
==========================================
  Files         336      336              
  Lines       22648    22651       +3     
  Branches     3560     3561       +1     
==========================================
- Hits        16317    16308       -9     
- Misses       4556     4569      +13     
+ Partials     1775     1774       -1     

Files with missing lines	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	83.83% <100.00%> (ø)
...iance/ComplianceIndexingOperationListenerImpl.java	64.86% <100.00%> (+1.48%)	⬆️
...security/configuration/DlsFlsFilterLeafReader.java	61.39% <100.00%> (ø)
...figuration/SecurityFlsDlsIndexSearcherWrapper.java	79.66% <0.00%> (-1.70%)	⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DarshitChanpura]

Great find and fix on this long present bug @cwperks !!