Floss Capa Refactor #2933

spoiicy · 2025-07-24T19:15:44Z

Description

This PR aims to update the Flare Capa and Flare Floss analyzers, by refactoring the exisiting implementation and utilizing the pip packages for better support.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review by using GitHub's reviewing system detailed here.

spoiicy · 2025-07-24T19:45:34Z

Capa Results

Floss Results

{
  "report": {
    "strings": {
      "stack_strings": [],
      "tight_strings": [],
      "static_strings": [],
      "decoded_strings": [],
      "language_strings": [],
      "language_strings_missed": []
    },
    "analysis": {
      "functions": {
        "library": 23,
        "discovered": 153,
        "analyzed_stack_strings": 104,
        "analyzed_tight_strings": 0,
        "analyzed_decoded_strings": 20,
        "decoding_function_scores": {
          "4198416": {
            "score": 0.6,
            "xrefs_to": 1
          },
          "4198640": {
            "score": 0.6,
            "xrefs_to": 1
          },
          "4198704": {
            "score": 0.6,
            "xrefs_to": 1
          },
          "4198848": {
            "score": 0.6,
            "xrefs_to": 1
          },
          "4203136": {
            "score": 0.607,
            "xrefs_to": 2
          },
          "4203440": {
            "score": 0.682,
            "xrefs_to": 10
          },
          "4203984": {
            "score": 0.571,
            "xrefs_to": 4
          },
          "4204592": {
            "score": 0.901,
            "xrefs_to": 72
          },
          "4204960": {
            "score": 0.714,
            "xrefs_to": 22
          },
          "4205664": {
            "score": 0.771,
            "xrefs_to": 2
          },
          "4206107": {
            "score": 0.693,
            "xrefs_to": 2
          },
          "4206162": {
            "score": 0.796,
            "xrefs_to": 25
          },
          "4206224": {
            "score": 0.7,
            "xrefs_to": 84
          },
          "4206869": {
            "score": 0.693,
            "xrefs_to": 2
          },
          "4207819": {
            "score": 0.732,
            "xrefs_to": 2
          },
          "4208097": {
            "score": 0.579,
            "xrefs_to": 4
          },
          "4208772": {
            "score": 0.85,
            "xrefs_to": 2
          },
          "4209782": {
            "score": 0.656,
            "xrefs_to": 2
          },
          "4210331": {
            "score": 0.745,
            "xrefs_to": 1
          },
          "4210406": {
            "score": 0.745,
            "xrefs_to": 1
          }
        }
      },
      "enable_stack_strings": true,
      "enable_tight_strings": true,
      "enable_static_strings": false,
      "enable_decoded_strings": true
    },
    "metadata": {
      "runtime": {
        "total": 6.5962,
        "vivisect": 3.2142,
        "start_date": "2025-07-22T04:05:41.936423Z",
        "find_features": 0.2296,
        "stack_strings": 0.2385,
        "tight_strings": 0.0003,
        "static_strings": 0,
        "decoded_strings": 2.6583,
        "language_strings": 0
      },
      "version": "3.1.1",
      "language": "unknown",
      "file_path": "/opt/deploy/files_required/06ebf06587b38784e2af42dd5fbe56e5",
      "imagebase": 4194304,
      "min_length": 4,
      "language_version": "version unknown",
      "language_selected": ""
    },
    "exceeded_max_number_of_strings": {}
  },
  "data_model": null,
  "errors": [],
  "parameters": {
    "rank_strings": {
      "stack_strings": true,
      "static_strings": false,
      "decoded_strings": false
    },
    "max_no_of_strings": {
      "stack_strings": 1000,
      "static_strings": 1000,
      "decoded_strings": 1000
    }
  }
}

spoiicy · 2025-07-24T19:53:03Z

Couple of points to note with this PR.

flare-capa pip package doesn't come with capa-rules and signatures by default unlike the binary file, so the update method dynamically pulls the latest versions for both.
flare-capa usually takes couple of seconds to analyze the given executable though when specifically tested on main.out test file from tests.zip, it takes enormous time to analyze (around 1000 seconds).
As a result, soft_time_limit parameter has been updated to 1800 seconds and a new timeout parameter has been added which can be set according to user's requirement.

drosetti · 2025-08-01T16:08:53Z

The code looks good to me, however I tried to replicate your analysis (main.out sample) but I had some problems with the updates.

Trying from the plugin pages I got an error message. In the logs there are these:

2025-08-01 16:07:22,183 - api_app.analyzers_manager.file_analyzers.capa_info - _unzip - INFO - Rules have been succesfully extracted
2025-08-01 16:07:22,184 - api_app.analyzers_manager.file_analyzers.capa_info - _download_signatures - INFO - Downloading signatures at /opt/deploy/files_required/capa/sigs now
2025-08-01 16:07:22,526 - api_app.analyzers_manager.file_analyzers.capa_info - _download_signatures - ERROR - Failed to download signature: Command '['/usr/bin/wget', '-O', '/opt/deploy/files_required/capa/sigs', 'https://raw.githubusercontent.com/mandiant/capa/master/sigs/1_flare_msvc_rtf_32_64.sig']' returned non-zero exit status 1.
2025-08-01 16:07:22,526 - api_app.analyzers_manager.file_analyzers.capa_info - update - ERROR - Failed to update capa rules with error: Failed to update signatures due to error: None

I tried to replicate the wget command from my command line and seems to work, am I missing something ?

spoiicy · 2025-08-01T17:47:21Z

Hi @drosetti I've made some fixes. It should work now.

fgibertoni

Nice work!

fgibertoni · 2025-08-04T06:20:45Z

api_app/analyzers_manager/file_analyzers/capa_info.py

+
+        for signature in signatures_list:
+            try:
+                subprocess.run(


Just to be extra sure, I'd like to make shlex handle splitting and quoting of parameters before passing them into run()

api_app/analyzers_manager/file_analyzers/capa_info.py

api_app/analyzers_manager/file_analyzers/floss.py

drosetti · 2025-08-05T09:20:10Z

@spoiicy now I can download correctly the rules, but when I run the analysis I get an error:
Analyzer for main.out failed with error: ERROR capa: [Errno 13] Permission denied: main.py:678\n '/opt/deploy/intel_owl/.cache/capa

mlodic · 2025-08-12T16:01:57Z

api_app/analyzers_manager/file_analyzers/capa_info.py

+                    capture_output=True,
+                )
+
+            except subprocess.CalledProcessError as e:


If possible I would like to prefer avoid using subprocess for a tool (wget) that performs HTTPS connections that can be handled with classic HTTP libraries. Can you change this?

fgibertoni

Please also update migration number since we merged the other PR. Then we should be good 😃

mlodic

good job! I asked to do additional work regarding the management of the rules which is critical in terms of performance and actual effectiveness of this complex analyzer. Please let me know if you have any doubts about this.

api_app/analyzers_manager/file_analyzers/capa_info.py

mlodic · 2025-08-25T06:57:58Z

api_app/analyzers_manager/migrations/0164_update_capa.py

+    AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig")
+
+    pm = PythonModule.objects.get(
+        module="capa_info.CapaInfo",


if I remember correctly there is also a param called "docker_based" that should be set to False in the migration to both the analyzers

api_app/analyzers_manager/file_analyzers/capa_info.py

mlodic · 2025-08-25T07:19:41Z

api_app/analyzers_manager/file_analyzers/capa_info.py

+        try:
+
+            response = requests.get(file_url, stream=True)
+            logger.info(f"Started downloading rules from {file_url}")


log the version

mlodic · 2025-08-25T07:19:48Z

api_app/analyzers_manager/file_analyzers/capa_info.py

+            logger.error(f"Failed to download rules with error: {e}")
+            raise AnalyzerRunException("Failed to download rules")
+
+        logger.info(f"Rules have been successfully downloaded at {RULES_LOCATION}")


log the version

api_app/analyzers_manager/file_analyzers/capa_info.py

mlodic · 2025-08-25T07:30:07Z

api_app/analyzers_manager/file_analyzers/capa_info.py

+
+    @classmethod
+    def _download_signatures(cls) -> None:
+        logger.info(f"Downloading signatures at {SIGNATURE_LOCATION} now")


the "signatures" are less important than the rules ones. These are almost never updated while the "rules" are are updated often. Plus, most of the time, we don't want these signatures to execute either cause it would slow the Capa execution. The rules are always necessary because they are the core part of the tool while these one could not be necessary. Because of that, I would not re-update them once they are here, like you already do. But we need another additional parameter for the user to enable them explicitly otherwise it would be better if these signatures would be disabled by default.

reference: https://github.com/mandiant/capa/tree/master/sigs

Plus, most of the time, we don't want these signatures to execute either cause it would slow the Capa execution.

Regarding your point, Actually I've tried executing the flare-capa without the signatures but it threw an error, when only executed with rules. So, I feel the signatures are necessary for it's execution.

Though, I can definitely make changes in the code that the signatures are only downloaded once or updated on-demand by the user.

made the changes such that signatures are only downloaded the first or whenever force_pull_signatures is set to True.

…oading of signatures

code-review-doctor

Looks good. Worth considering though. View full project report here.

code-review-doctor · 2025-09-05T21:32:55Z

api_app/analyzers_manager/models.py

+
+
+class AnalyzerRulesFileVersion(models.Model):
+    last_downloaded_version = models.CharField(max_length=50, blank=True, null=True)


Suggested change

last_downloaded_version = models.CharField(max_length=50, blank=True, null=True)

last_downloaded_version = models.CharField(max_length=50, blank=True, default="")

null=True on a string field causes inconsistent data types because the value can be either str or None. This adds complexity and maybe bugs, but can be solved by replacing null=True with default="". More info.

code-review-doctor · 2025-09-05T21:32:55Z

api_app/analyzers_manager/models.py

+
+class AnalyzerRulesFileVersion(models.Model):
+    last_downloaded_version = models.CharField(max_length=50, blank=True, null=True)
+    download_url = models.URLField(max_length=200, blank=True, null=True)


Suggested change

download_url = models.URLField(max_length=200, blank=True, null=True)

download_url = models.URLField(max_length=200, blank=True, default="")

Similarly, consider replacing null=True with default="" (and blank=True to pass validation checks).

fgibertoni

Greatly improved! I'll let @mlodic take a look at this to see if it's how he expected to be 😃

fgibertoni · 2025-09-08T06:28:47Z

api_app/analyzers_manager/migrations/0166_update_capa.py

+    p1 = Parameter(
+        name="timeout",
+        type="float",
+        description="Duration in seconds for which intelowl waits for capa to return results. Default is set to 15 seconds.",


Maybe I am missing something, but have you set the default of the new parameter?

You are right, I've not set the default for this one. No worries I'll update and add it. :)

Perfect, thank you!

spoiicy marked this pull request as ready for review July 24, 2025 20:09

spoiicy requested a review from fgibertoni July 31, 2025 13:50

fgibertoni requested a review from drosetti August 1, 2025 14:35

spoiicy force-pushed the floss_capa_refactor branch from 9400e1c to ca4d856 Compare August 1, 2025 17:05

fgibertoni requested changes Aug 4, 2025

View reviewed changes

mlodic reviewed Aug 12, 2025

View reviewed changes

spoiicy requested a review from mlodic August 23, 2025 10:01

fgibertoni reviewed Aug 24, 2025

View reviewed changes

spoiicy force-pushed the floss_capa_refactor branch from 767c04d to 918965d Compare August 24, 2025 17:41

mlodic requested changes Aug 25, 2025

View reviewed changes

mlodic mentioned this pull request Aug 25, 2025

Gsoc25 refactor analyzer tests #2886

Merged

23 tasks

Akshit Maheshwary added 10 commits September 6, 2025 02:57

refactored floss analyzer to utilize pip package

5639151

refactored capa_info to utilize pip package

87b8e89

added timeout parameter and changed soft time limit

63a5226

removed docker related sections and updated requirements

fd200af

fixed deepsource issues

6978938

fixed deepsource issues

b93481a

fixed wget command

da6eb26

shell-escaped user input and switched to requests lib to handle downl…

f4bc110

…oading of signatures

updated migration numbers

f06c049

tracking rules_version and minor changes

c57d234

spoiicy force-pushed the floss_capa_refactor branch from 751159c to c57d234 Compare September 5, 2025 21:32

code-review-doctor bot suggested changes Sep 5, 2025

View reviewed changes

fixed deepsource issue and updated migration numbers

045bc30

spoiicy force-pushed the floss_capa_refactor branch from ddeecc4 to 045bc30 Compare September 6, 2025 03:11

fixed capa and floss tests

2370697

spoiicy requested review from mlodic and fgibertoni September 7, 2025 06:01

fgibertoni reviewed Sep 8, 2025

View reviewed changes



		class AnalyzerRulesFileVersion(models.Model):
		last_downloaded_version = models.CharField(max_length=50, blank=True, null=True)

	download_url = models.URLField(max_length=200, blank=True, null=True)
	download_url = models.URLField(max_length=200, blank=True, default="")

Uh oh!

Floss Capa Refactor #2933

Are you sure you want to change the base?

Floss Capa Refactor #2933

Uh oh!

Conversation

spoiicy commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Important Rules

Uh oh!

spoiicy commented Jul 24, 2025

Capa Results

Floss Results

Uh oh!

spoiicy commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Couple of points to note with this PR.

Uh oh!

drosetti commented Aug 1, 2025

Uh oh!

spoiicy commented Aug 1, 2025

Uh oh!

fgibertoni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

drosetti commented Aug 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fgibertoni left a comment

Choose a reason for hiding this comment

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

code-review-doctor bot left a comment

Choose a reason for hiding this comment

Uh oh!

code-review-doctor bot Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

code-review-doctor bot Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

fgibertoni left a comment

Choose a reason for hiding this comment

Uh oh!

spoiicy commented Jul 24, 2025 •

edited

Loading

spoiicy commented Jul 24, 2025 •

edited

Loading

spoiicy Sep 8, 2025 •

edited

Loading