“Chat with RTX” is an application that allows customization of a GPT large language model (LLM) with your specific content like documents and notes. This enables you to generate contextually relevant answers from a bespoke chatbot. The entire setup operates locally on RTX-enabled Windows PCs or workstations, ensuring quick and secure access to information.

I have an RTX 4060Ti and Windows 11 desktop. Below shows the steps to set it up.

Requirement

• GPU NVIDIA GeForce™ RTX 30 or 40 Series GPU or NVIDIA RTX™ Ampere or Ada Generation GPU with at least 8GB of VRAM
• RAM 16GB or greater
• OS Windows 11
• Driver 535.11 or later
• File Size 35 GB

link : https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/

Setup

1. Install visual studio
2. Download cuda toolkit
3. Install driver in toolkit. Reboot.
4. Choose customize installation, install toolkit only. Disable :

   - GeForce Experience Software
   - PhysX
   - Driver (alreadly installed)
   

   Example of output: ``` Installed:

   • Nsight for Visual Studio 2022
   • Nsight Monitor Not Installed:
   • Nsight for Visual Studio 2019 Reason: VS2019 was not found
   • Nsight for Visual Studio 2017 Reason: VS2017 was not found
   • Integrated Graphics Frame Debugger and Profiler Reason: see https://developer.nvidia.com/nsight-vstools
   • Integrated CUDA Profilers Reason: see https://developer.nvidia.com/nsight-vstools ```
5. Install CUDNN

   Download cudnn for windows (cuda 12.x).
   Copy files to the nvidia tool kit folder (include/bin/lib files).
   Reboot
   

6. Install ChatWithRTX

   Download ChatWithRTX
   Run setup, select clean installation (it takes around 30-60 mins ...)
   To run it, search "nvidia chat with rtx", select open