Tag Archives: LLM

AI, GPU, Linux, LLMs, Tools

Installing ComfyUI with Python 3.12 on Debian 13 (Trixie) with CUDA

This guide provides instructions for installing and configuring ComfyUI on Debian 13 (Trixie) using Python 3.12. The process encompasses system preparation, Python version management, dependency installation, and configuration for optimal performance with NVIDIA GPU support.

The installation assumes that NVIDIA graphics hardware and CUDA are properly installed and configured on the system. For users who need guidance on setting up CUDA specifically for Debian 13 (Trixie), a related tutorial is available at: Building Llama.cpp with CUDA on Debian 13 (Trixie).

Prerequisites and System Preparation

Before initiating the ComfyUI installation process, it is crucial to ensure that the system has all necessary dependencies installed. This foundational step involves updating the package repository and installing development tools and libraries required for building and running the ComfyUI application effectively.

The initial system preparation begins with updating the package list to access the latest available packages:

sudo apt update

Following this update, a comprehensive set of build tools and libraries must be installed. These dependencies are fundamental for compiling software, managing Python environments, and supporting graphical operations that ComfyUI requires:

sudo apt install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev git gcc bc

In addition to the core development dependencies, several system-level packages are essential for proper functionality. These include utilities for managing Python virtual environments, graphics libraries for rendering, and core system libraries:

sudo apt install wget git python3 python3-venv libgl1 libglib2.0-0

These packages establish the necessary foundation for Python version management, Git operations, and graphical interface support that ComfyUI requires for optimal performance.

Installing Python 3.12 Using pyenv

ComfyUI requires Python 3.12 for full compatibility with its latest features and performance optimizations. Since Debian 13 (Trixie) may not include this specific Python version in its default repositories, we utilize pyenv to manage the installation and execution of the required Python environment.

The installation process begins with downloading and executing the official pyenv installation script from the pyenv repository:

curl https://pyenv.run | bash

This command fetches and executes the installation script, setting up the pyenv environment in the user’s home directory. Following the installation, proper shell configuration is essential to initialize pyenv correctly for each terminal session.

The configuration involves appending specific environment variable exports to the .bashrc file. These settings ensure that pyenv is properly initialized and that the appropriate Python version paths are included in the system’s PATH:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
source ~/.bashrc

With the environment properly configured, the specific Python version can be installed using pyenv. The command below installs Python 3.12.12, which is compatible with ComfyUI requirements:

pyenv install 3.12.12

Creating and Configuring the ComfyUI Environment

After establishing the Python environment, the next step involves creating a dedicated directory for ComfyUI and setting up the project structure. This organization ensures proper isolation of dependencies and facilitates easy management of the installation.

The creation of the ComfyUI directory and navigation into it follows these commands:

mkdir ComfyUI
cd ComfyUI

To ensure that the correct Python version is used for this specific project, set the local Python version to 3.12.12 using pyenv:

pyenv local 3.12.12

This command creates a .python-version file in the current directory, which pyenv will automatically use when entering this directory in future sessions.

With the environment properly configured, the next step involves installing the ComfyUI command-line interface tool. This utility simplifies the installation and management of ComfyUI components:

pip install comfy-cli

Following the installation of the CLI tool, it is recommended to install shell completion support for enhanced usability:

comfy --install-completion

The final step in the initial setup process involves installing all necessary ComfyUI dependencies and components:

comfy install

This command downloads and configures all required packages and models, which may take considerable time depending on network speed and system resources.

Configuring CUDA Support

For users with NVIDIA graphics hardware, configuring CUDA support is essential for optimal performance. The installation process checks for the presence of CUDA by verifying the nvcc compiler version.

To determine if CUDA is properly installed, execute the following command:

nvcc --version

If CUDA is correctly installed, the output will display information similar to:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

If CUDA is detected, install the appropriate PyTorch version with CUDA support using the following command:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

The cu124 suffix corresponds to the CUDA compilation tools release 12.4, as shown in the example output. This ensures that PyTorch is compiled with support for the installed CUDA version, enabling GPU acceleration for ComfyUI operations.

Launching ComfyUI

With all dependencies properly installed and configured, ComfyUI can be launched using the command-line interface. The basic launch command starts the application with default settings:

comfy launch

For users who require remote access to the ComfyUI interface, the application can be configured to listen on all network interfaces and specific ports. This configuration enables access from other machines on the network:

comfy launch -- --listen 0.0.0.0 --port 8080

This command configures ComfyUI to accept connections from any IP address (0.0.0.0) on port 8080, making it accessible across the network while maintaining security through proper firewall configuration.

It is important to always ensure that you are working within the ComfyUI directory before launching the application. This practice guarantees that the correct Python version and dependencies are used, preventing potential conflicts or errors during execution.

With these steps completed, ComfyUI is successfully installed and configured to run with Python 3.12 on Debian 13 (Trixie). The system is now ready for use with NVIDIA graphics hardware and CUDA support, providing users with a powerful and flexible interface for creating complex image generation workflows.

AI, GPU, Linux, LLMs, Tools

Installing Stable Diffusion WebUI with Python 3.10 on Debian 13 (Trixie)

This guide provides detailed instructions for installing and configuring the Stable Diffusion WebUI on Debian 13 (Trixie), utilizing Python 3.10. The process involves several key steps including system preparation, Python version management, repository cloning, and configuration adjustments to enable network accessibility.

The installation assumes that NVIDIA graphics hardware and CUDA are already properly installed and configured on the system. For users who need guidance on setting up CUDA specifically for Debian 13 (Trixie), a related tutorial is available at: Building Llama.cpp with CUDA on Debian 13 (Trixie).

Prerequisites and System Preparation

Before beginning the installation process, it is essential to ensure that the system has all necessary dependencies installed. This includes development tools and libraries required for building and running the Stable Diffusion WebUI application.

The first step involves updating the package list to ensure access to the latest available packages. This is followed by installing a comprehensive set of build tools and libraries that are fundamental for compiling software and managing Python environments:

sudo apt update
sudo apt install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev git gcc bc

In addition to the core development dependencies, several system-level packages are required for proper functionality. These include utilities for managing Python virtual environments, graphics libraries for rendering, and core system libraries:

sudo apt install wget git python3 python3-venv libgl1 libglib2.0-0

These packages provide the foundation necessary for Python version management, Git operations, and graphical interface support that the Stable Diffusion WebUI requires.

Installing Python 3.10 Using pyenv

The Stable Diffusion WebUI specifically requires Python 3.10, which may not be available in the default repositories for Debian 13 (Trixie). To address this requirement, we utilize pyenv, a powerful tool for managing multiple Python versions on a single system.

The installation of pyenv begins with downloading and executing the official installation script from the pyenv repository:

curl https://pyenv.run | bash

This command fetches the installation script and executes it, setting up the pyenv environment in the user’s home directory. Following the installation, it is necessary to configure the shell environment to properly initialize pyenv each time a new terminal session is started.

The configuration involves appending specific environment variable exports to the .bashrc file. These settings ensure that pyenv is correctly initialized and that the appropriate Python version paths are included in the system’s PATH:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
source ~/.bashrc

Once the environment variables are properly configured, the specific Python version can be installed using pyenv. The command below installs Python 3.10.6, which is compatible with the Stable Diffusion WebUI requirements:

pyenv install 3.10.6

Cloning and Configuring the Stable Diffusion WebUI Repository

With the Python environment properly established, the next step involves obtaining the source code for the Stable Diffusion WebUI. This is accomplished by cloning the official repository from GitHub, which contains all necessary files and dependencies for running the web interface.

The cloning process retrieves the complete repository including all branches, commit history, and configuration files:

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui

After successfully cloning the repository, navigate into the newly created directory. This is where all subsequent configuration and setup operations will take place:

cd stable-diffusion-webui

To ensure that the correct Python version is used for this specific project, set the local Python version to 3.10.6 using pyenv:

pyenv local 3.10.6

This command creates a .python-version file in the current directory, which pyenv will automatically use when entering this directory in future sessions.

Launching the WebUI Application

With all prerequisites met and the environment properly configured, the final step involves starting the Stable Diffusion WebUI application. This is accomplished by executing the webui.sh script, which handles the initialization process including dependency installation and server startup:

webui.sh

The execution of this script may take some time as it downloads required model files and dependencies, initializes the Python environment, and prepares the web server for operation. Users should allow sufficient time for this process to complete fully.

Configuring Network Accessibility

By default, the Stable Diffusion WebUI is configured to only accept connections from the local machine. For users who wish to access the interface from other devices on the network, a configuration change is necessary.

The configuration file webui-user.sh contains various settings that can be adjusted to modify the behavior of the web application. To enable network accessibility, this file must be edited:

nano webui-user.sh

Within this file, locate the line that begins with #export COMMANDLINE_ARGS="". This line is commented out by default and serves as a placeholder for additional command-line arguments. To modify the behavior to accept external connections, change this line to:

export COMMANDLINE_ARGS="--listen"

This configuration change instructs the web application to listen on all available network interfaces rather than restricting access to localhost only. This modification enables remote access to the Stable Diffusion WebUI from other machines within the same network. The interface can be reached at http://<server-ip>:7860

With these comprehensive steps completed, the Stable Diffusion WebUI is successfully installed and configured to run with Python 3.10 on Debian 13 (Trixie). The system is now ready for use with NVIDIA graphics hardware and CUDA support, providing users with a fully functional interface for generating images using stable diffusion models.

AI, GPU, How-tos, Linux, LLMs, Tools

Building llama.cpp with CUDA on Debian 13 “Trixie”

2 Comments

If you’ve recently upgraded to Debian 13 or are fresh on a Trixie system, you may be eager to tap the power of your NVIDIA GPU for machine‑learning workloads. This post walks you through every step required to set up the necessary drivers, libraries, and build environment.

Why Enable CUDA in llama.cpp?

The original binaries of llama.cpp run on the CPU, which is perfectly fine for small models but can become a bottleneck with larger weights. By enabling the -DGGML_CUDA=ON flag, the project compiles the CUDA kernels that allow your NVIDIA GPU to perform inference. The result is a dramatic reduction in latency and a higher throughput for text generation tasks.

Prerequisites

  • A Debian 13 machine with an NVIDIA GPU that supports CUDA 11 or later.
  • Sudo access (or root) to install packages and modify system configuration.
  • An active internet connection so the package manager can fetch the necessary files.

Step 1 – Update Kernel Headers

Your system needs the headers that match the running kernel so that the NVIDIA driver can compile its kernel modules.

apt install linux-headers-$(uname -r)

This command pulls the headers for the current kernel release and installs them into the standard package locations.

Step 2 – Add Non‑Free Firmware Repositories

The Debian base repositories do not expose the proprietary firmware and driver packages needed for NVIDIA GPUs. By creating an additional source list file, we allow apt to pull the required non‑free components.

Create the file /etc/apt/sources.list.d/non‑free.sources and paste the following content:

Types: deb deb-src
URIs: http://deb.debian.org/debian/
Suites: trixie
Components: non-free-firmware contrib
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb deb-src
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: non-free-firmware contrib
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb deb-src
URIs: http://deb.debian.org/debian/
Suites: trixie-updates
Components: non-free-firmware contrib
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

After saving the file, refresh the package lists so the new entries become available:

apt update

Step 3 – Install the NVIDIA Driver and CUDA Toolkit

3.1 Bring in the NVIDIA Keyring

The NVIDIA distribution for Debian ships a keyring package that allows your system to verify the authenticity of the driver packages.

wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb

3.2 Install Driver Packages

apt -V install nvidia-driver-cuda nvidia-kernel-dkms

The meta‑package nvidia-driver-cuda pulls the latest driver binaries and the CUDA toolkit for the current kernel. It also installs nvidia-kernel-dkms, which provides a Dynamic Kernel Module Support interface so the driver can be built against any future kernel version.

3.3 Regenerate Initramfs and Update GRUB

After installing the driver modules, you must ensure that the initramfs contains the new driver and that GRUB will boot into the updated kernel configuration.

update-initramfs -u -k all
update-grub

Reboot the machine to let the new driver take effect.

3.4 Install the CUDA Toolkit

With the driver in place, install the toolkit components that provide nvcc, libraries, and headers used by llama.cpp.

apt install nvidia-cuda-toolkit

Step 4 – Install Build Dependencies

The build process for llama.cpp requires several libraries and developer tools. Installing them up front keeps the compile step straightforward.

apt install libtcmalloc-minimal4 libcurl4-openssl-dev glslc cmake make git pkg-config

These packages provide memory allocation utilities, SSL support, the GLSL compiler, CMake, Make, Git, and generic build configuration tools.

Step 5 – Clone and Compile llama.cpp

With the environment prepared, fetch the source code and build it.

cd ~
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake .. \
  -DGGML_AVX=ON \
  -DGGML_AVX_VNNI=ON \
  -DGGML_AVX2=ON \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON
make -j8
echo 'export PATH=$PATH:'$(realpath bin) >> ~/.bashrc

After the build finishes, log out and back in again so the newly added binaries become visible in your shell path.

Step 6 – Keep the Driver in Sync with Kernel Updates

Kernel upgrades are common, and the driver must be rebuilt against each new kernel. The following routine ensures the driver modules stay current.

apt install linux-headers-$(uname -r)
apt install --reinstall nvidia-driver-cuda nvidia-kernel-dkms
apt install nvidia-cuda-toolkit
update-initramfs -u -k all
update-grub

Running this sequence after any kernel upgrade guarantees that the driver continues to load correctly.

Step 7 – Updating the Source Tree

When the upstream llama.cpp project publishes a new release or a bug fix, refresh your local copy and rebuild:

cd ~
cd llama.cpp/

# Clean the working directory
git clean -xdf
mkdir build

# Pull the latest changes and submodules
git pull
git submodule update --recursive

# Rebuild
cd build/
cmake .. \
  -DGGML_AVX=ON \
  -DGGML_AVX_VNNI=ON \
  -DGGML_AVX2=ON \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON
make -j8
AI, GPU, How-tos, Linux, LLMs

Running Large Language Models on Cheap Old RX 580 GPUs with llama.cpp and Vulkan

2 Comments

LLMs and GPUs

In recent years, the landscape of artificial intelligence has shifted dramatically with the rise of large language models (LLMs). These models are incredibly powerful but also resource-intensive — typically requiring high-end GPUs like NVIDIA’s RTX 4090s or AMD’s latest Radeon Instinct series to run effectively.

But what if you don’t have access to such hardware? What if your budget is limited, or you already own older GPUs like the AMD Radeon RX 580? Surprisingly, there’s still a way to get meaningful performance out of these aging cards — especially with the right software stack and a bit of ingenuity.

This guide walks through how to leverage the AMD Radeon RX 580 — an aging yet capable GPU — to run large language models using llama.cpp via Vulkan API support, even though ROCm (the newer AMD compute framework) no longer supports it.

Hardware Overview: The Radeon RX 580

The Radeon RX 580 is part of AMD’s Polaris generation, released in 2016. While not cutting-edge today, it still offers:

  • 8 GB GDDR5 memory (sufficient for many smaller models)
  • 2,304 stream processors
  • 14nm process
  • Good PCIe 3.0 bandwidth

Although it’s no longer officially supported in newer versions of ROCm, the RX 580 retains full compatibility with Vulkan drivers, making it ideal for running modern AI inference engines.

Software Stack: llama.cpp + Vulkan

llama.cpp is a lightweight C++ implementation of the LLaMA architecture that allows you to run LLMs directly on your CPU or GPU.

It supports multiple backends including:

  • CPU (default)
  • CUDA (NVIDIA)
  • Metal (Apple Silicon)
  • Vulkan (AMD & Intel GPUs)

By enabling Vulkan support during compilation, we can tap into the RX 580’s full potential.

Installing Vulkan Drivers on Debian 12

Before we build llama.cpp, we need to ensure the system has proper Vulkan support:

sudo apt install vulkan-tools libtcmalloc-minimal4 libcurl4-openssl-dev glslc cmake make git pkg-config libvulkan-dev

These packages provide:

  • vulkan-tools: Tools for testing Vulkan applications
  • libtcmalloc-minimal4: Memory allocator for performance
  • libcurl4-openssl-dev: For downloading models via HTTP
  • glslc: GLSL shader compiler (needed for Vulkan)
  • cmake, make, git, pkg-config: Build dependencies
  • libvulkan-dev: Required for Vulkan development

Once installed, you can verify Vulkan support:

vulkaninfo | grep -i RX

You should see your GPU listed in the output.

Installing llama.cpp with Vulkan Support

Let’s walk through the full installation process.

Step 1: Clone the Repository

cd ~
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

Step 2: Configure CMake for Vulkan

Build llama.cpp with Vulkan enabled:

cmake .. \
  -DGGML_AVX=ON \
  -DGGML_AVX_VNNI=ON \
  -DGGML_AVX2=ON \
  -DGGML_VULKAN=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON

This configuration enables:

  • AVX instructions for faster CPU ops
  • AVX2 / VNNI optimizations (for better performance on supported CPUs)
  • Vulkan backend support for AMD GPUs
  • Curl support for downloading GGUF models from Hugging Face

Step 3: Compile and Install

make -j8
echo 'export PATH=$PATH:'$(realpath bin) >> ~/.bashrc

Log out and back in to update your environment variables so llama-cli and llama-server are available in your terminal.

Running Models with llama-cli and llama-server

Now that everything is built, let’s test it out with some sample commands.

Using llama-cli

Run a model using the CLI interface:

llama-cli -m deepseek-r1:8B --device Vulkan0 -ngl 99

This command:

  • Loads a model named deepseek-r1:8B
  • Uses device Vulkan0 (first Vulkan-compatible GPU detected)
  • Sets -ngl 99 to offload all layers to GPU

You can optionally specify the full model path or use Hugging Face URLs (with the -hf flag if supported).

Using llama-server

To expose your model via an API endpoint:

llama-server --host 0.0.0.0 -hf unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_M --device Vulkan0 -ngl 99

This starts a server listening on all interfaces (0.0.0.0) and uses:

  • unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_M as the model (quantized to 4-bit)
  • Device Vulkan0
  • All layers (-ngl 99) loaded into GPU memory

Multi-GPU Setup

If you have more than one RX 580 (or other Vulkan-compatible GPUs), you can split the model across multiple devices:

llama-server --host 0.0.0.0 -hf unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q8_K_XL --device Vulkan0,Vulkan1

And for even larger models, like Qwen3-Coder-30B-A3B-Instruct-GGUF:

llama-server \
  --host 0.0.0.0 \
  -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q8_K_XL \
  -ngl 99 \
  --threads -1 \
  --ctx-size 32684 \
  --temp 0.7 \
  --min-p 0.0 \
  --top-p 0.80 \
  --top-k 20 \
  --repeat-penalty 1.05 \
  --device Vulkan0,Vulkan1,Vulkan2,Vulkan3,Vulkan4

This will use up to five GPUs, distributing load across them and enabling inference of 30B parameter models.

Updating llama.cpp

When new updates are released, just run:

cd ~/llama.cpp/
git clean -xdf
git pull
git submodule update --recursive
cd build/
cmake .. \
  -DGGML_AVX=ON \
  -DGGML_AVX_VNNI=ON \
  -DGGML_AVX2=ON \
  -DGGML_VULKAN=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON
make -j8

Performance Notes: RX 580 Limitations and Workarounds

While the RX 580 isn’t the fastest GPU on the market, it can still run impressive models when properly configured. Here are some key takeaways:

  • Small to medium-sized models (e.g., 7B–13B parameters) run smoothly with minimal latency.
  • Larger models (like 30B) require:
  • Quantized weights (Q4, Q8_K_XL)
  • Multi-GPU setup
  • Longer wait times for responses
  • Threading optimization (--threads -1)
  • Higher context sizes (--ctx-size)

Despite limitations, a cluster of 5 RX 580s can handle a 30B parameter model, which is quite remarkable for such older hardware.

Final Thoughts

The RX 580 may be old, but it still holds value in the world of AI inference. Thanks to the llama.cpp project’s Vulkan backend support, it’s possible to run large language models on low-cost hardware that would otherwise be unusable for AI workloads.

With careful configuration and the right software stack, you can build a capable local LLM inference rig using nothing more than a few secondhand GPUs. Whether you’re training, experimenting, or just curious about AI, this setup provides a great foundation to get started.

If you’re looking to repurpose an old rig or build a cost-effective edge AI box, the RX 580 + Vulkan + llama.cpp combination is worth exploring — and you might be surprised at what it can do.

Have questions or need help setting up your own RX 580-based LLM cluster? Leave a comment below or share your experience in the comments!