A beginner's guide to deploying LLMs with AMD on Windows using PyTorch

Summary

If you’re interested in deploying advanced AI models on your local hardware, leveraging a modern AMD GPU or APU can provide an efficient and scalable solution. You don’t need dedicated AI infrastructure to experiment with Large Language Model (LLMs); a capable Microsoft® Windows® PC with PyTorch installed and equipped with a recent AMD graphics card is all you need. PyTorch for AMD on Windows and Linux is now available as a public preview. You can now use native PyTorch for AI inference on AMD Radeon™ RX 7000 and 9000 series GPUs and select AMD Ryzen™ AI 300 and AI Max APUs, enabling seamless AI workload execution on AMD hardware in Windows without any need for workarounds or dual-boot configurations. If you are new and just getting started with AMD ROCm™, be sure to check out our getting started guides here. This guide is designed for developers seeking to set up, configure, and execute LLMs locally on a Windows PC using PyTorch with an AMD GPU or APU. No previous experience with PyTorch or deep learning frameworks is needed. What you’ll need (the prerequisites) The currently supported AMD platforms and hardware for PyTorch on Windows are listed here: Part 1: Setting up your workspace Step 1: Open the Command Prompt First, we need to open the Command Prompt Click the Start Menu, type cmd, and press Enter. A black terminal window will pop up. Step 2: Create and activate a virtual environment A “virtual environment” is like a clean, empty sandbox for a Python project. In your Command Prompt, type the following command and press Enter. This creates a new folder named llm-pyt that will house our project. python -m venv llm-pyt Next, we need to “activate” this environment. Think of this as stepping inside the sandbox. llm-pyt\Scripts\activate You’ll know it worked because you’ll see (llm-pyt) appear at the beginning of your command line prompt. Step 3: Install PyTorch and other essential libraries Now we’ll install the software libraries that do the heavy lifting. The m...