Handle invalid UTF-8 characters

[juli1]

What problem are you trying to solve?

We want to be able to scan files with encoding different than UTF-8

What is your solution?

Change the functions to read files so that

• we try to read UTF-8 first
• if we fail, we fallback and convert the invalid UTF-8 characters into artifacts

Testing

Added integration tests with the sast-files-encoding repository that exposes data with different encodings.

[datadog-datadog-prod-us1]

Datadog Summary

✅ Code Quality    ✅ Code Security    ✅ Dependencies

[amaanq]

This looks like it's stacked off of #676, if so, for reviewing purposes can you please set the target branch to your other feature branch, until that PR is merged? I was very confused at first as to why a lot of the code was the same here.

[juli1]

This looks like it's stacked off of #676, if so, for reviewing purposes can you please set the target branch to your other feature branch, until that PR is merged? I was very confused at first as to why a lot of the code was the same here.

Interesting since I use a different and clean branch. Will revert the other changes for sure.

[amaanq]

approved but undo the seemingly accidental newline change to integration-test-secrets.sh