Hacker Remix

Scalene: A high-performance, high-precision CPU, GPU, memory profiler for Python

178 points by kristianp 2 days ago | 21 comments

infocollector 1 day ago

I've used Scalene at various times in the past few years, and always liked using it when I want to dig deeper compared to cProfile/profile. You might also want to look at:

1] https://github.com/joerick/pyinstrument

2] https://github.com/benfred/py-spy

3] https://github.com/P403n1x87/austin

4] https://github.com/bloomberg/memray

5] https://github.com/pyutils/line_profiler

1st1 1 day ago

For profiling memory consider far more advanced memray.

https://github.com/bloomberg/memray

codedokode 21 hours ago

In my experience, memory profilers that show line or function which allocated memory, are not very useful. What I usually want instead is to get a graph of all objects in memory with their size and path from root, to see who is using most memory. To see for example, that there is some cache or log that is never cleared.

For PHP and for browser's developer tools there is such kind of profiler. But judging by screenshots, this profiler cannot produce such a graph. So its usability to solve memory leak/high consumption issues is limited.

To summarize, what I usually need is not a line number, but a path to objects using/holding most memory.

PathOfEclipse 17 hours ago

You are comparing two different types of things:

1. Allocation profiler

2. Heap analyzer

Allocation profilers will capture data about what is allocating memory over time. This can be captured in real time without interrupting the process and is usually relatively low-overhead.

Heap analyzers will generally take a heap dump, construct an object graph, do various analyses, and generate an interactive report. This generally requires that you pause a program long enough to create a heap dump, which is often multiple GB or more in size, write it to disk, then do the subsequent analysis and report generation.

I agree that 2) is generally more useful but I assume both types of profilers have their place and purpose.

Loranubi 1 day ago

Note: The Windows version currently only supports CPU and GPU profiling, but not memory or copy profiling.

That's a problem with many of the profiling tools around Python. They often support Windows badly or not at all.

solarkraft 1 day ago

Isn’t the modern Windows runtime just Linux anyway?

pjc50 19 hours ago

Er .. no? The modern Windows runtime, for various definitions of those words, is the confusingly named "WinRT" https://learn.microsoft.com/en-us/windows/uwp/cpp-and-winrt-...

Underneath it's still substantially similar to good old Windows NT.

There's a Linux "subsystem". Well, two of them. WSL1 is an API translation layer that ends up being cripplingly slow. Don't use it. WSL2 is more of a VM that just runs a Linux distro. This is before you get into third party compatibility layers like cygwin and mingw.

emeryberger 1 day ago

(Scalene author here) Nope, but WSL2 (Windows Subsystem for Linux) is, and Scalene works great with it.