Skip to content

gh-130695: add count create/destroy refs to tracemalloc #130696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Doc/library/tracemalloc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,8 @@ Functions

.. function:: clear_traces()

Clear traces of memory blocks allocated by Python.
Clear traces of memory blocks allocated and counts of created and destroyed
memory blocks by Python.
Comment on lines +302 to +303
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Clear traces of memory blocks allocated and counts of created and destroyed
memory blocks by Python.
Clear traces of memory blocks allocated by Python, and reset counts
of created and destroyed references.


See also :func:`stop`.

Expand Down Expand Up @@ -330,6 +331,12 @@ Functions
:mod:`tracemalloc` module as a tuple: ``(current: int, peak: int)``.


.. function:: get_traced_allocs()

Get the current unit counts of allocated and deallocated memory blocks.
:mod:`tracemalloc` module as a tuple: ``(allocs: int, deallocs: int)``.


.. function:: reset_peak()

Set the peak size of memory blocks traced by the :mod:`tracemalloc` module
Expand Down
9 changes: 9 additions & 0 deletions Include/internal/pycore_tracemalloc.h
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ struct _tracemalloc_runtime_state {
/* domain (unsigned int) => traces (_Py_hashtable_t).
Protected by TABLES_LOCK(). */
_Py_hashtable_t *domains;
/* Number of allocations.
Protected by TABLES_LOCK(). */
Py_ssize_t allocations;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to rename it to "ref_created" and "ref_destroyed".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started with that but as per colesbury above: """The terminology here is a bit confusing, but I don't think we should be calling these "references" or "refs". It's tracking object allocation and deallocation, not references or reference counts.""". Technically its tracking the first reference on an object (not all references), so excluding some unallocated "static" objects, its essentially allocations, but its called RefTracer, so that's unfortunate. But maybe following the established naming convention, although imperfect, is better? Anyways, putting it back to refs_created/_destroyed, but of course you have the final say so let me know which you prefer.

/* Number of deallocations.
Protected by TABLES_LOCK() and sometimes modified atomically. */
Py_ssize_t deallocations;

struct tracemalloc_traceback empty_traceback;

Expand Down Expand Up @@ -155,6 +161,9 @@ extern size_t _PyTraceMalloc_GetMemory(void);
/* Get the current size and peak size of traced memory blocks as a 2-tuple */
extern PyObject* _PyTraceMalloc_GetTracedMemory(void);

/* Get the current number of references created and destroyed as a 2-tuple */
extern PyObject* _PyTraceMalloc_GetTracedAllocs(void);

/* Set the peak size of traced memory blocks to the current size */
extern void _PyTraceMalloc_ResetPeak(void);

Expand Down
32 changes: 32 additions & 0 deletions Lib/test/test_tracemalloc.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import textwrap
import tracemalloc
import unittest
import warnings
from unittest.mock import patch
from test.support.script_helper import (assert_python_ok, assert_python_failure,
interpreter_requires_environment)
Expand Down Expand Up @@ -1141,6 +1142,37 @@ def __del__(self, untrack=_testcapi.tracemalloc_untrack):
""")
assert_python_ok("-c", code)

def test_trace_refs(self):
def f():
l = []
del l

def g():
l = [], []
del l

tracemalloc.start()

try:
tracemalloc.clear_traces()
f()
refs = tracemalloc.get_traced_allocs()
if refs == (1, 0):
warnings.warn("ceval Py_DECREF doesn't emit PyRefTracer_DESTROY in this build")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect a test failure, not a warning. In which case we would fall into this case? And why ignoring this failure?

else:
self.assertEqual(refs, (1, 1))

tracemalloc.clear_traces()
g()
refs = tracemalloc.get_traced_allocs()
if refs == (3, 2):
warnings.warn("ceval Py_DECREF doesn't emit PyRefTracer_DESTROY in this build")
else:
self.assertEqual(refs, (3, 3))

finally:
tracemalloc.stop()


if __name__ == "__main__":
unittest.main()
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add tracking of number of deallocations to :mod:`tracemalloc`.
18 changes: 18 additions & 0 deletions Modules/_tracemalloc.c
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,23 @@ _tracemalloc_get_traced_memory_impl(PyObject *module)
return _PyTraceMalloc_GetTracedMemory();
}


/*[clinic input]
_tracemalloc.get_traced_allocs

Get the current count of created and destroyed refs.

Returns a tuple: (created: int, destroyed: int).
[clinic start generated code]*/

static PyObject *
_tracemalloc_get_traced_allocs_impl(PyObject *module)
/*[clinic end generated code: output=cb84cbd781fab85f input=336f38d37a5830ef]*/
{
return _PyTraceMalloc_GetTracedAllocs();
}


/*[clinic input]
_tracemalloc.reset_peak

Expand Down Expand Up @@ -195,6 +212,7 @@ static PyMethodDef module_methods[] = {
_TRACEMALLOC_GET_TRACEBACK_LIMIT_METHODDEF
_TRACEMALLOC_GET_TRACEMALLOC_MEMORY_METHODDEF
_TRACEMALLOC_GET_TRACED_MEMORY_METHODDEF
_TRACEMALLOC_GET_TRACED_ALLOCS_METHODDEF
_TRACEMALLOC_RESET_PEAK_METHODDEF
/* sentinel */
{NULL, NULL}
Expand Down
22 changes: 21 additions & 1 deletion Modules/clinic/_tracemalloc.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions Python/tracemalloc.c
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ typedef struct {
#define tracemalloc_tracebacks _PyRuntime.tracemalloc.tracebacks
#define tracemalloc_traces _PyRuntime.tracemalloc.traces
#define tracemalloc_domains _PyRuntime.tracemalloc.domains
#define tracemalloc_allocations _PyRuntime.tracemalloc.allocations
#define tracemalloc_deallocations _PyRuntime.tracemalloc.deallocations


#ifdef TRACE_DEBUG
Expand Down Expand Up @@ -1258,6 +1260,10 @@ _PyTraceMalloc_TraceRef(PyObject *op, PyRefTracerEvent event,
void* Py_UNUSED(ignore))
{
if (event != PyRefTracer_CREATE) {
/* we don't want bother here with the lock for performance reasons */
if (_Py_atomic_load_int_relaxed(&tracemalloc_config.tracing)) {
_Py_atomic_add_ssize(&tracemalloc_deallocations, 1);
}
return 0;
}
if (get_reentrant()) {
Expand All @@ -1271,6 +1277,8 @@ _PyTraceMalloc_TraceRef(PyObject *op, PyRefTracerEvent event,
goto done;
}

tracemalloc_allocations += 1;

PyTypeObject *type = Py_TYPE(op);
const size_t presize = _PyType_PreHeaderSize(type);
uintptr_t ptr = (uintptr_t)((char *)op - presize);
Expand Down Expand Up @@ -1326,6 +1334,8 @@ _PyTraceMalloc_ClearTraces(void)
TABLES_LOCK();
if (tracemalloc_config.tracing) {
tracemalloc_clear_traces_unlocked();
tracemalloc_allocations = 0;
tracemalloc_deallocations = 0;
}
TABLES_UNLOCK();
}
Expand Down Expand Up @@ -1464,6 +1474,25 @@ _PyTraceMalloc_GetTracedMemory(void)
return Py_BuildValue("nn", traced, peak);
}


PyObject *
_PyTraceMalloc_GetTracedAllocs(void)
{
TABLES_LOCK();
Py_ssize_t allocations, deallocations;
if (tracemalloc_config.tracing) {
allocations = tracemalloc_allocations;
deallocations = tracemalloc_deallocations;
}
else {
allocations = 0;
deallocations = 0;
}
TABLES_UNLOCK();

return Py_BuildValue("nn", allocations, deallocations);
}

void
_PyTraceMalloc_ResetPeak(void)
{
Expand Down
Loading