dynamic: render: ValueError "name not found for call"

[mike-hunhoff]

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/spring/Documents/capa/capa/main.py", line 1149, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/spring/Documents/capa/capa/main.py", line 1054, in main
    print(capa.render.vverbose.render(meta, rules, capabilities.matches))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 540, in render
    return render_vverbose(rd.ResultDocument.from_capa(meta, rules, capabilities))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 533, in render_vverbose
    render_rules(console, doc)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 516, in render_rules
    render_match(console, doc.meta.analysis.layout, rule, match, indent=1)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 332, in render_match
    render_match(console, layout, rule, child, indent=indent + 1, mode=child_mode)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 329, in render_match
    render_node(console, layout, rule, match, match.node, indent=indent)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 280, in render_node
    render_statement(console, layout, match, node.statement, indent=indent)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 182, in render_statement
    render_locations(console, layout, match.locations, indent)
  File "/home/spring/Documents/capa/capa/render/vverbose.py", line 104, in render_locations
    s = f"{v.render_call(layout, location)}\nand {(len(locations) - 1)} more..."
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spring/Documents/capa/capa/render/verbose.py", line 150, in render_call
    cname = _get_call_name(layout, addr)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spring/Documents/capa/capa/render/verbose.py", line 112, in _get_call_name
    raise ValueError("name not found for call", addr)
ValueError: ('name not found for call', Address(type=<AddressType.CALL: 'call'>, value=(1476, 2652, 1948, 52762)))

[mike-hunhoff]

Sample is not public. My initial triage shows that not all matched calls are recorded in matched_calls even though thread and processes are recorded as expected.

[mike-hunhoff]

This doesn't seem specific to a particular process/thread combination. I'm seeing missing entries across the board.

[mike-hunhoff]

Ah it looks like part of the problem is

capa/capa/loader.py

Line 584 in 151d30b

calls_by_thread[t.address] = []

which assumes a thread is only recorded once per process. In my example (VMRay archive), threads appear to be recorded more than once per process. I'm not sure if this is a bug in the VMRay feature extractor, or expected behavior that needs to be handled above...headed there next.

[mike-hunhoff]

I've confirmed that VMRay is tracking duplicate thread IDs for a single process, which makes sense as TIDs can be recycled by the underlying operating system. This is related to the PID reuse discussion in #2361. Basically, we need to make the process and thread address classes more unique, using information provided by the underlying sandbox, or, generating our own when parsing.

[dhruvak001]

If i am not wrong the error occurs because CAPA assumes thread addresses (TIDs) are unique per process, but OS-level TID reuse causes collisions in dynamic analysis data.

We can change the code in loader.py mentioned above to:
thread_key = (p.address.value, t.address.value, t.start_time)
calls_by_thread[thread_key] = []

Is this okay to proceed with @mike-hunhoff ?

[williballenthin]

there are at least two further considerations:

1. every time a thread (or call) is referenced, the thread start time (or other uniquing factor) must also be used, so there will be other code changes needed too.

2. we should keep in mind how other dynamic backends, like CAPE, will be affected by these changes, and ideally updated, too. e.g., does CAPE have the thread start time available?

this will take a little investigation and discussion, and you're welcomed to research this further @dhruvak001. if you do, please start by reproducing locally and developing unit test cases so we are all working towards precisely the same goal.