- Look up the CUDA Programming Guide from NVidia

- CUDA Programming books from NVidia from developer.nvidia.com/cuda-books-archive link

- Start creating small programs based on the existing implementations (A strong C implementation knowledge is required. So, brush up if needed.)

- Install the required Toolchains, compilers, and I am assuming you have the necessary hardware to play around

- Github links with CUDA projects. Read the code, And now you could use LLM to explain the code in the way you would need

- Start creating smaller, yet parallel programs etc., etc.,

And in about a month or two, you should have enough to start writing CUDA programs.

I'm not aware of the skill / experience levels you have, but whatever it might be, there are plenty of sources and resources available now than it was in 2007/08.

Create a 6-8 weeks of study plan and you should be flying soon!

Hope it helps.

Feel free to comment and I can share whatever I could to guide.

CUDA itself is just a minor departure from C++, so the language itself is no big deal if you've used C++ before. But, if you're trying to get hired programming CUDA, what that really means is they want you implementing AI stuff (unless it's game dev). AI programming is a much wider and deeper subject than CUDA itself, so be ready to spend a bunch of time studying and hacking to come up to speed in that. But if you do, you will be in high demand. As mentioned, the fast.ai videos are a great introduction.

In the case of games, that means 3D graphics which these days is another rabbit hole. I knew a bit about this back in the day, but it is fantastically more sophisticated now and I don't have any idea where to even start.

The real money is in mastering PTX, nvcc, cuobjdump, Nsight Systems, and Nsight Compute. CUTLASS is good open source code base to explore - start here https://christianjmills.com/series/notes/cuda-mode-notes.htm...

most importantly, stay off HN, get on Discord gpu mode, where real coders are: https://discord.com/invite/gpumode

https://www.gpumode.com/ resources and discord community Book: Programming massively parallel processors nvidia cuda docs are very comprehensive too https://github.com/srush/GPU-Puzzles

1. Learning CUDA - the framework, libraries and high-layer wrappers. This is something that changes with times and trends.

2. Learning high-performance computing approaches. While a GPU and the Nvlink interfaces are Nvidia specific, working in a massively-parallel distributed computing environment is a general branch of knowledge that is translatable across HPC architectures.

3. Application specifics. If your thing is Transformers, you may just as well start from Torch, Tensorflow, etc. and rely on the current high-level abstractions, to inspire your learning down to the fundamentals.

I’m no longer active in any of the above, so I can’t be more specific, but if you want to master CUDA, I would say learning how massive-parallel programming works, is the foundation that may translate into transferable skills.

Disclaime: I don't claim that this is actually a systematic way to learn it and it is more for academic work.

I got assigned to a project that needs learning CUDA as part of my PhD. There was no one in my research group who have any experience or know CUDA. I started with standard NVIDIA courses (Getting Started with Accelerated Computing with CUDA C/C++ and there is python version too).

This gave me good introduction to the concepts and basic ideas but I think after that I did most of learning by trial and error. I tried a couple of online tutorials for specific things and some books but it was always a deprecated function there or here or a change of API that make things obsolete. Or basically things changed for your GPU and now you have to be careful because yoy might be using GPU version not compatible with what I develop for in production and you need things to work for both.

I think learning CUDA for me is an endeavor of pain and going through "compute-sanitizer" and Nsight because you will find that most of your time will go into debugging why things is running slower than you think.

Take things slowly. Take a simple project that you know how to do without CUDA then port it to CUDA ane benchmark against CPU and try to optimize different aspect of it.

The one advice that can be helpful is not to think about optimization at the beginning. Start with correct, then optimize. A working slow kernel beats a fast kernel that corrupts memory.

Make sure you are very clear on what you want. Most HR departments cast a wide net (it's like how every junior role requires "3-5 years of experience" when in reality they don't really care). Similarly when hiring, most companies pray for the unicorn developer who can understand the entire stack from the GPU to the end user product domain when the day to day is mostly in Python.

Scientific Parallel Computing by L. Ridgway Scott et. al. - https://press.princeton.edu/books/hardcover/9780691119359/sc...

https://developer.download.nvidia.com/compute/cuda/2_2/sdk/w...

After this you should be able to tell whether you enjoy this kind of work.

If you do, try to do a reasonably optimized GEMM, and then try to follow the FlashAttention paper and implement a basic version of what they're doing.

Do not implement algorithms by hand. Recent architectures are extremely hard to reach decent occupancy and such. Thrust and cub solve 80% of the cases with reasonable trade-offs and they do most of the work for you.

https://developer.nvidia.com/thrust

The YouTube Channel - CUDA_MODE - it is based on PMPP I could not find the channel, but here is the playlist https://www.youtube.com/watch?v=LuhJEEJQgUM&list=PLVEjdmwEDk...

Once done, you would be on solid foundation.

https://gfxcourses.stanford.edu/cs149/fall24/lecture/datapar...

So - start getting used to programming without using for loops, would be my tip.

https://developer.nvidia.com/gpugems/gpugems3/part-v-physics...

As an Apple platforms developer I actually worked through those books to figure out how to convert the CUDA stuff to Metal, which helped the material click even more.

Part of why I did it was – and this was some years back – I wanted to sharpen my thinking around parallel approaches to problem solving, given how central those algorithms and ways of thinking are to things like ML and not just game development, etc.

https://www.youtube.com/playlist?list=PLxNPSjHT5qvtYRVdNN1yD...

After watching this video I was able to implement a tiling version of a kernel that was the bottleneck of our production data analysis pipeline to improve performance by over 2x. There's much more to learn but I found this video series to be a great place to start.

I found it easy to start. Then there was a pretty nice learning curve to get to warps, SM's and basic concepts. Then I was able to dig deeper into the integer opcodes, which was super cool. I was able to optimize the compute part pretty well, without much roadblocks.

However, getting memory loads perfect and then getting closer to hw (warp groups, divergence, the L2 cache split thing, scheduling), was pretty hard.

I'd say CUDA is pretty nice/fun to start with, and it's possible to get quite far for a novice programmer. However getting deeper and achieving real advantage over CPU is hard.

Additionally there is a problem with Nvidia segmenting the market - some opcodes are present in _old_ gpu's (CUDA arch is _not_ forwards compatible). Some opcodes are reserved to "AI" chips (like H100). So, to get code that is fast on both H100 and RTX5090 is super hard. Add to that a fact that each card has different SM count and memory capacity and bandwidth... and you end up with an impossible compatibility matrix.

TLDR: Beginnings are nice and fun. You can get quite far on the optimizing compute part. But getting compatibility for differnt chips and memory access is hard. When you start, chose specific problem, specific chip, specific instruction set.

Start writing some CUDA core to sort an array or find the maximum element.

That doesn't mean one-eyed-king knowledge is never enough to solve that chicken-and-egg. You only have to be good enough to get the job.

But if you haven't done it on the job, you don't have work experience and you are either lying to others or lying to yourself...and any sophisticated organization won't fall for it...

...except of course, knowingly. And the best way to get someone to knowingly ignore obvious dunning-kruger and/or horseshit is to know that someone personally or professionally.

Which is to say that the best way to get a good job is to have a good relationship with someone who can hire you for a good job (nepotism trumps technical ability, always). And the best way to find a good job is to know a lot of people who want to work with you.

To put it another way, looking for a job is the only way to find a job and looking for a job is also much much harder than everything that avoids looking for a job (like studying CUDA) by pretending to be preparation...because again, studying CUDA won't ever give you professional experience.

Don't get me wrong, there's nothing wrong with learning CUDA all on your own. But it is not professional experience and it is not looking for a job doing CUDA.

Finally, if you want to learn CUDA just learn it for its own sake without worrying about a job. Learning things for their own sake is the nature of learning once you get out of school.

Good luck.