Skip to content

Commit f56506f

Browse files
mlodiccristinaascaridependabot[bot]0ssigeno
authored
Merge pull request #2663 from intelowlproject/develop
* Fix vt hash (#2654) * fix vt3 and added debug logs * fix file read * Bump quark-engine from 24.12.1 to 25.1.1 in /requirements (#2650) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.12.1 to 25.1.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.12.1...v25.1.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 4.1.0 to 4.2.2 (#2652) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.0 to 4.2.2. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.1.0...v4.2.2) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump mobsfscan in /integrations/malware_tools_analyzers/requirements (#2634) Bumps [mobsfscan](https://github.com/MobSF/mobsfscan) from 0.3.9 to 0.4.5. - [Release notes](https://github.com/MobSF/mobsfscan/releases) - [Commits](MobSF/mobsfscan@0.3.9...0.4.5) --- updated-dependencies: - dependency-name: mobsfscan dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump django-ses from 4.2.0 to 4.3.1 in /requirements (#2643) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.2.0 to 4.3.1. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.2.0...v4.3.1) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ossf/scorecard-action from 2.3.0 to 2.4.0 (#2648) Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.0 to 2.4.0. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@483ef80...62b2cac) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump thug in /integrations/malware_tools_analyzers/requirements (#2644) Bumps [thug](https://github.com/buffer/thug) from 4.9 to 6.10. - [Release notes](https://github.com/buffer/thug/releases) - [Commits](buffer/thug@v4.9...v6.10) --- updated-dependencies: - dependency-name: thug dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * reverting Thug and adding disclaimer * More logs + fix investigation status on job removal (#2653) * More logs + fix investigation status on job removal * Retry insert if path already exists * Fix datamodel generic job serializer (#2662) * fix job serializer data model generic * fix job serializer data model generic * Bump fangfrisch in /integrations/malware_tools_analyzers/requirements (#2660) Bumps [fangfrisch](https://github.com/rseichter/fangfrisch) from 1.6.0 to 1.9.0. - [Release notes](https://github.com/rseichter/fangfrisch/releases) - [Changelog](https://github.com/rseichter/fangfrisch/blob/master/CHANGELOG.rst) - [Commits](rseichter/fangfrisch@1.6.0...1.9.0) --- updated-dependencies: - dependency-name: fangfrisch dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/upload-artifact from 4.3.1 to 4.5.0 (#2651) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.3.1 to 4.5.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@5d5d22a...6f51ac0) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump elasticsearch-dsl from 8.15.0 to 8.17.0 in /requirements (#2646) Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.15.0 to 8.17.0. - [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases) - [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst) - [Commits](elastic/elasticsearch-dsl-py@v8.15.0...v8.17.0) --- updated-dependencies: - dependency-name: elasticsearch-dsl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github/codeql-action from 2.22.11 to 3.28.0 (#2645) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.11 to 3.28.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Commits](github/codeql-action@v2.22.11...v3.28.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gunicorn in /integrations/malware_tools_analyzers/requirements (#2638) Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 22.0.0 to 23.0.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@22.0.0...23.0.0) --- updated-dependencies: - dependency-name: gunicorn dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gunicorn from 22.0.0 to 23.0.0 in /integrations/tor_analyzers (#2637) Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 22.0.0 to 23.0.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@22.0.0...23.0.0) --- updated-dependencies: - dependency-name: gunicorn dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gunicorn from 22.0.0 to 23.0.0 in /integrations/pcap_analyzers (#2633) Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 22.0.0 to 23.0.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@22.0.0...23.0.0) --- updated-dependencies: - dependency-name: gunicorn dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-python from 5.1.0 to 5.3.0 (#2632) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.1.0 to 5.3.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5.1.0...v5.3.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bump and changelog --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Cristina Ascari <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <[email protected]>
2 parents 0949248 + fdc28bb commit f56506f

File tree

19 files changed

+90
-43
lines changed

19 files changed

+90
-43
lines changed

.github/CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22

33
[**Upgrade Guide**](https://intelowlproject.github.io/docs/IntelOwl/installation/#update-to-the-most-recent-version)
44

5+
## [v6.2.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.2.1)
6+
Minor fixes and dependencies upgrades
7+
58
## [v6.2.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.2.0)
69

710
Happy new year! :sparkler: And Happy Birthday IntelOwl! :tada:

.github/workflows/codeql-analysis.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,14 +37,14 @@ jobs:
3737

3838
steps:
3939
- name: Checkout repository
40-
uses: actions/checkout@v4.1.0
40+
uses: actions/checkout@v4.2.2
4141
with:
4242
# We must fetch at least the immediate parents so that if this is
4343
# a pull request then we can checkout the head.
4444
fetch-depth: 2
4545

4646
- name: Set up Python
47-
uses: actions/setup-python@v5.1.0
47+
uses: actions/setup-python@v5.3.0
4848
with:
4949
python-version: '3.11'
5050

@@ -65,7 +65,7 @@ jobs:
6565
6666
# Initializes the CodeQL tools for scanning.
6767
- name: Initialize CodeQL
68-
uses: github/codeql-action/init@v3
68+
uses: github/codeql-action/init@v3.28.0
6969
with:
7070
languages: python
7171
# Override the default behavior so that the action doesn't attempt
@@ -93,4 +93,4 @@ jobs:
9393
# make release
9494

9595
- name: Perform CodeQL Analysis
96-
uses: github/codeql-action/analyze@v3
96+
uses: github/codeql-action/analyze@v3.28.0

.github/workflows/dependency_review.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,6 @@ jobs:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- name: 'Checkout Repository'
14-
uses: actions/checkout@v4.1.0
14+
uses: actions/checkout@v4.2.2
1515
- name: 'Dependency Review'
1616
uses: actions/dependency-review-action@v4

.github/workflows/pull_request_automation.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@ jobs:
1616
outputs:
1717
frontend: ${{steps.diff_check.outputs.frontend}}
1818
steps:
19-
- uses: actions/checkout@v4.1.0
19+
- uses: actions/checkout@v4.2.2
2020
with:
2121
ref: ${{ github.base_ref }}
22-
- uses: actions/checkout@v4.1.0
22+
- uses: actions/checkout@v4.2.2
2323
with:
2424
clean: false
2525
- name: Generate diffs
@@ -34,10 +34,10 @@ jobs:
3434

3535
steps:
3636
- name: Checkout IntelOwl
37-
uses: actions/checkout@v4.1.0
37+
uses: actions/checkout@v4.2.2
3838

3939
- name: Set up Python
40-
uses: actions/setup-python@v5.1.0
40+
uses: actions/setup-python@v5.3.0
4141
with:
4242
python-version: 3.11
4343

@@ -69,7 +69,7 @@ jobs:
6969
access_token: ${{ github.token }}
7070

7171
- name: Checkout IntelOwl
72-
uses: actions/checkout@v4.1.0
72+
uses: actions/checkout@v4.2.2
7373

7474
- name: Prepare Launch
7575
run: |
@@ -119,7 +119,7 @@ jobs:
119119
if: ${{ needs.detect-changes.outputs.frontend > 0 }}
120120

121121
steps:
122-
- uses: actions/checkout@v4.1.0
122+
- uses: actions/checkout@v4.2.2
123123
- name: Set up NodeJS
124124
uses: actions/setup-node@v4
125125
with:

.github/workflows/scorecard.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,12 @@ jobs:
3232

3333
steps:
3434
- name: "Checkout code"
35-
uses: actions/checkout@v4.1.0 # v3.1.0
35+
uses: actions/checkout@v4.2.2 # v3.1.0
3636
with:
3737
persist-credentials: false
3838

3939
- name: "Run analysis"
40-
uses: ossf/scorecard-action@483ef80eb98fb506c348f7d62e28055e49fe2398 # v2.3.0
40+
uses: ossf/scorecard-action@62b2cac7ed8198b15735ed49ab1e5cf35480ba46 # v2.4.0
4141
with:
4242
results_file: results.sarif
4343
results_format: sarif
@@ -59,7 +59,7 @@ jobs:
5959
# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
6060
# format to the repository Actions tab.
6161
- name: "Upload artifact"
62-
uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
62+
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
6363
with:
6464
name: SARIF file
6565
path: results.sarif
@@ -68,6 +68,6 @@ jobs:
6868

6969
# Upload the results to GitHub's code scanning dashboard.
7070
- name: "Upload to code-scanning"
71-
uses: github/codeql-action/upload-sarif@03e7845b7bfcd5e7fb63d1ae8c61b0e791134fab # v2.22.11
71+
uses: github/codeql-action/upload-sarif@5b6e617dc0241b2d60c2bccea90c56b67eceb797 # v2.22.11
7272
with:
7373
sarif_file: results.sarif

api_app/investigations_manager/models.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1+
import logging
12
from datetime import datetime
23
from typing import List
34

45
from django.conf import settings
56
from django.db import models
7+
from django.db.models import QuerySet
68

79
from api_app.choices import TLP
810
from api_app.interfaces import OwnershipAbstractModel
@@ -11,8 +13,11 @@
1113
from api_app.models import ListCachable
1214
from certego_saas.apps.user.models import User
1315

16+
logger = logging.getLogger(__name__)
17+
1418

1519
class Investigation(OwnershipAbstractModel, ListCachable):
20+
jobs: QuerySet
1621
name = models.CharField(max_length=100)
1722
description = models.TextField(default="", blank=True)
1823

@@ -60,13 +65,18 @@ def user_can_edit(self, user: User) -> bool:
6065
def set_correct_status(self, save: bool = True):
6166
from api_app.models import Job
6267

68+
logger.info(f"Setting status for investigation {self.pk}")
6369
# if I have some jobs
6470
if self.jobs.exists():
6571
# and at least one is running
6672
for job in self.jobs.all():
6773
job: Job
6874
jobs = job.get_tree(job)
69-
if jobs.exclude(status__in=Job.STATUSES.final_statuses()).count() > 0:
75+
running_jobs = jobs.exclude(status__in=Job.STATUSES.final_statuses())
76+
if running_jobs.count() > 0:
77+
logger.info(
78+
f"Jobs {running_jobs.values_list('pk', flat=True)} are still running for investigation {self.pk}"
79+
)
7080
self.status = self.STATUSES.RUNNING.value
7181
self.end_time = None
7282
break

api_app/investigations_manager/views.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,10 +102,13 @@ def remove_job(self, request, pk):
102102
investigation.refresh_from_db()
103103
# we are possibly changing the status of the investigation
104104
investigation.set_correct_status(save=True)
105-
return Response(
105+
response = Response(
106106
status=status.HTTP_200_OK,
107107
data=InvestigationSerializer(instance=investigation).data,
108108
)
109+
if not investigation.jobs.exists():
110+
investigation.delete()
111+
return response
109112

110113
@action(methods=["GET"], url_name="graph", detail=True)
111114
def tree(self, request, pk):

api_app/mixins.py

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,11 +104,25 @@ def _perform_request(
104104
self, uri: str, method: str, ignore_404: bool = False, **kwargs
105105
) -> Dict:
106106
error = None
107+
response = None
107108
try:
108109
url = self.url + uri
110+
logger.debug(f"kwargs: {kwargs}")
109111
if method == "GET":
110-
response = requests.get(url, headers=self.headers, **kwargs)
112+
if kwargs:
113+
logger.debug(
114+
f"about to send get request to url {url} with headers {self.headers} and kwargs: {kwargs}"
115+
)
116+
response = requests.get(url, headers=self.headers, **kwargs)
117+
else:
118+
logger.debug(
119+
f"about to send get request to url {url} with headers {self.headers} and no kwargs"
120+
)
121+
response = requests.get(url, headers=self.headers)
111122
elif method == "POST":
123+
logger.debug(
124+
f"about to send post request to url {url} with headers {self.headers} and kwargs: {kwargs}"
125+
)
112126
response = requests.post(url, headers=self.headers, **kwargs)
113127
else:
114128
raise NotImplementedError()
@@ -121,6 +135,9 @@ def _perform_request(
121135
# .. it happens when a requested object is not found and that's normal
122136
if not ignore_404 or not response.status_code == 404:
123137
response.raise_for_status()
138+
except requests.exceptions.JSONDecodeError as e:
139+
error_message = f"Raised JSONDecodeError: {e}. Error data: {response.text if response is not None else None}"
140+
raise AnalyzerRunException(error_message)
124141
except Exception as e:
125142
error_message = f"Raised Error: {e}. Error data: {error}"
126143
raise AnalyzerRunException(error_message)
@@ -414,7 +431,9 @@ def _vt_scan_file(self, md5: str, rescan_instead: bool = False) -> Dict:
414431
else:
415432
logger.info(f"(Job: {self.job_id}, {md5}) -> VT analyzer requested scan")
416433
try:
434+
self._job.file.seek(0)
417435
binary = self._job.file.read()
436+
logger.debug(f"BINARY: {binary}")
418437
except Exception:
419438
raise AnalyzerRunException(
420439
"IntelOwl error: couldn't retrieve the binary"
@@ -444,7 +463,8 @@ def _vt_scan_file(self, md5: str, rescan_instead: bool = False) -> Dict:
444463
)
445464
for chance in range(max_tries):
446465
time.sleep(poll_distance)
447-
result, _ = self._perform_get_request(uri, files=files)
466+
result, _ = self._perform_get_request(uri)
467+
logger.debug(f"result: {result}")
448468
analysis_status = (
449469
result.get("data", {}).get("attributes", {}).get("status", "")
450470
)

api_app/models.py

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -562,14 +562,13 @@ def set_final_status(self) -> None:
562562
"finished_analysis_time",
563563
]
564564
)
565-
try:
566-
# we update the status of the analysis
567-
if root_investigation := self.get_root().investigation:
568-
root_investigation.set_correct_status(save=True)
569-
except Exception as e:
570-
logger.exception(
571-
f"investigation status not updated. Job: {self.pk}. Error: {e}"
572-
)
565+
# we update the status of the analysis
566+
if root_investigation := self.get_root().investigation:
567+
from api_app.investigations_manager.models import Investigation
568+
569+
logger.info(f"Updating status of investigation {root_investigation.pk}")
570+
root_investigation: Investigation
571+
root_investigation.set_correct_status(save=True)
573572

574573
def __get_config_reports(self, config: typing.Type["AbstractConfig"]) -> QuerySet:
575574
return getattr(self, f"{config.__name__.split('Config')[0].lower()}reports")

api_app/queryset.py

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,13 @@
1515
from treebeard.mp_tree import MP_NodeQuerySet
1616

1717
if TYPE_CHECKING:
18-
from api_app.models import PythonConfig, AbstractConfig
18+
from api_app.models import PythonConfig
1919
from api_app.serializers import AbstractBIInterface
2020

2121
import logging
2222

2323
from celery.canvas import Signature
24-
from django.db import models
24+
from django.db import IntegrityError, models
2525
from django.db.models import (
2626
BooleanField,
2727
Case,
@@ -280,7 +280,16 @@ def create(self, parent=None, **kwargs):
280280
"""
281281
if parent:
282282
return parent.add_child(**kwargs)
283-
return self.model.add_root(**kwargs)
283+
try:
284+
return self.model.add_root(**kwargs)
285+
except IntegrityError as e:
286+
if "path" in str(e):
287+
logger.warning(
288+
f"Found race condition for {kwargs['name']}. Trying again to calculate path."
289+
)
290+
# we try again a second time, hoping for no race condition
291+
return self.model.add_root(**kwargs)
292+
raise
284293

285294
def delete(self, *args, **kwargs):
286295
"""

0 commit comments

Comments
 (0)