 |
|
|
|
Accounting for memory use in a program is often difficult. The TfMallocTag system is designed to track memory use in a hierarchical fashion. Memory use in this context refers specifically to heap allocations that you make using malloc() (and its variants) and free(). Note that this includes the C++ new and delete operators, since they (in general) simply call through to malloc() and free() (but see Best Profiling Results as well).
The basic idea is that at any point during program execution, you can push a memory "tag" onto the local call stack. Allocations made while that tag is active are "billed" to that tag. Then later you can query and see how much outstanding memory is due to that particular tag. If additional tags are pushed onto the stack, memory allocations are billed to these "children" tags but are also included in the bill to the parent tags.
Each line of code in the program that pushes a tag is called a call-site. A sequence of call-sites is called a path node and describes the hierarchy under which allocations take place. You use an object (TfAutoMallocTag) whose constructor pushes the tag and whose destructor pops the tag to control the lifetime of a tag. Consider the following example:
Suppose now that you invoke TopFunction(). The program has three different call-sites, Top, A and B. However, running TopFunction() generates the following distinct path nodes: Top, Top/A, Top/A/B, and Top/B.
The total memory billed to path node Top is 400. Calling TopFunction() results in allocations in itself, its direct calls FuncA() and FuncB(), and its indirect call to FuncB() from FuncA(). The direct memory billed to the path node Top is simply 100. That is the memory allocation noted by the line in the example marked note1. Even though this call comes after FuncA() has been called, the call-site tag A is no longer active, since it was popped off the stack when FuncA() exited.
Continuing the analysis, the total memory billed to Top/A is 200, the memory billed to Top/A/B is 100, and the memory billed to Top/B is also 100. To access these statistics, you call the GetCallTree() function on the TfMallocTag object. Note that the system does not begin any actual accounting until Initialize() is called. Any memory allocations that occur prior to this point are "off the radar."
The following example shows a typical use of tags in a library. Note that memory tags in a program should be distinct to avoid unintended aliasing, so follow the same basic guidelines that apply to avoiding name-conflicts in functions and classes.
So far, all the tags shown in examples have been string literals. Sometimes it is useful to construct tag names on the fly, as in the following example:
This technique can generate any number of different call-sites (each distinct name passed in generates a different call-site) and thus a whole sequence of different path nodes.
Unless a dynamic string provides substantial value (as including the model-name may above), prefer a string literal or a TfEternalString (such as the one returned by TF_FUNC_NAME()) where you have the choice. Either of these is substantially cheaper, because the implementation can key on the string's address rather than its characters. A std::string built on the fly does not get that fast path: it always resolves by content, and additionally costs the construction of the string itself on every call, whether or not tagging is active. Extremely performance-intensive code should pass a string literal or a TfEternalString; failing that, a previously constructed string is still better than building one on every call.
Be aware also that a call site passing dynamic names can generate a very large number of distinct call sites and path nodes, each of which is retained for the life of the process. A handful of such sites can dominate the total path node count. If you find yourself generating unbounded distinct names, consider whether a fixed set of tags with the varying part omitted would answer your question.
Occasionally, using local variables to delimit the scope of a tag isn't possible. You can make manual calls to Push() and Pop(), but whenever possible use TfAutoMallocTag.
TfMallocTag::GetTotalBytes(), TfMallocTag::GetMaxTotalBytes() and TfMallocTag::GetCallTree() all report on allocations that are currently live.
Everything these calls return is a snapshot. On a running multithreaded program, allocation activity continues while the snapshot is being assembled and immediately after it is returned, so treat the numbers as accurate to a moment that has already passed rather than as an instantaneous truth.
TfMallocTag::GetMaxTotalBytes() is a high-water mark sampled at each internal consolidation, rather than a continuous one. The background consolidator wakes on every buffer spill and on a periodic timer (see Configuration), and is additionally forced by the queries above, so sampling is frequent – and it is densest when allocation is most active, since spill rate is what drives it. Even so, a spike that rises and falls entirely within one consolidation interval is not observed.
TfMallocTag::Clear() resets all accounting and counters while leaving the system running, so subsequent reports cover only allocations made after the call. TfMallocTag::Shutdown() stops tracking altogether and returns overhead to its lowest level; call Initialize() again to resume, and note that anything allocated while shut down is untracked thereafter.
TfMallocTag::PauseControl is a scoped object that suppresses attribution of allocations, either on the calling thread (PauseThisThread()) or across all threads (PauseAllThreads()). Collection resumes when the object is destroyed. This is useful for excluding a region whose allocations would otherwise obscure what you are trying to measure.
Note that pausing suppresses the reported bytes, not the underlying bookkeeping cost. A paused allocation is still tracked internally, because the eventual free has to be matched against something. So PauseControl is a tool for focusing collection on areas of interest, not for making a hot region cheap.
Programs that have tags but have never called Initialize() have no measurable increase in running time: pushing and popping a tag is an inline read of a global and a branch.
Collection works by interposing the allocator, which requires ArchMallocHook. When unavailable, PXR_ARCH_SUPPORT_MALLOC_HOOKS is left undefined and TfMallocTag compiles as stubs: Initialize() always fails, the queries report no memory, and a tag scope compiles to nothing. Calling code needs no changes, and reports captured elsewhere can still be loaded and printed with CallTree::LoadReport() and CallTree::Report().
With tagging active, the system intercepts every malloc() and free() in the process, so the cost lands on whatever code is allocating. For typical workloads, measured overhead has been around 10 percent of total wall-time. A program that does nothing but allocate and free could see substantially more.
Where that cost goes, in rough terms:
The system also reserves address space for its internal tables; see Configuration.
TfMallocTag::GetPerfStats() reports on the tagging system's own internal behavior. It is intended for people working on TfMallocTag itself rather than for users of it.
The memory statistics can be fooled by programs which contain their own allocators: in this case, the memory requested by the allocator itself does not correlate well with what tags are active. A pool allocator that grabs a large block under one tag and then hands pieces of it out under many others will bill the entire block to the first tag. It is best to turn off as many internal pooling allocators as possible when profiling.
TfMallocTag attributes an allocation to whatever tag was active at the moment malloc() was called, which is exactly what you want when the allocation is made on behalf of the work you tagged, and misleading when a cache or a pool defers or batches it. When a result looks wrong, suspect this first.
The TfMallocTag system is thread-safe. Each thread has its own tag stack, so tags pushed on one thread do not affect attribution on another. TfMallocTag::GetCallTree() and TfMallocTag::GetTotalBytes() may be called at any time and from any thread, subject to the cost described in Reading Results.
Because each thread has its own stack, work handed to a task scheduler is by default billed to whatever tags happen to be active on the worker thread that picks it up, which is usually not what you want. TfMallocTag::StackState and TfMallocTag::StackOverride exist to bridge that gap: a call site that spawns parallel tasks can capture its own tag stack with TfMallocTag::GetCurrentStackState() and pass it to the tasks, each of which installs it with a scoped StackOverride so its allocations are billed to the spawning site's tags.
Three environment settings control how much memory TfMallocTag uses internally. None normally needs changing.
PXR_TF_MALLOC_TAG_EVENT_BUFFERS_MB (default 16) sizes the pool of recycled event-collection buffers, reused to absorb spills without malloc/free churn. This bounds only the recycle pool; each live thread also owns a buffer it is currently filling, and spilled buffers awaiting consolidation are not counted against this limit, so total event-buffer memory under load can exceed it. PXR_TF_MALLOC_TAG_CONSOLIDATE_PERIOD_MS (default 23) sets the period in milliseconds between opportunistic wakes of the background consolidator. On each periodic wake, the consolidator drains all threads so that slowly-allocating or stalled threads contribute to the running total and high-water sampling while their data is still current. A value of 0 disables periodic waking. PXR_TF_MALLOC_TAG_PATH_NODE_TABLE_MB (default 128) sizes the hash table mapping path nodes, rounded down to a power-of-two number of buckets. This is address space, not resident memory: only the buckets actually touched are committed, so a program that creates few tags pays little of it. Raise it if TfMallocTag warns that the table is overloaded during memory profiling. An overloaded table simply performs suboptimally; it does not suffer data loss. 
1.12.0