1

What I want?

I want to buy a GPU for amateurly using and fine-tuning LLMs or LMMs. I don't have enough money for buying RTX4090. I can buy RTX4060 TI 16GB VRAM or RTX4070 12GB VRAM. But i am so confused.

Why I am confused?:

When I tried to use LLMs on a 15 VRAM GPU azure cloud instance with vLLM docker images, So many LLM didn't work because VRAM not enough errors thrown from CUDA. I understood VRAM too important for using LLM with CUDA. RTX4060 slower but have more VRAM, RTX4070 faster but have less VRAM. Actually I don't need high speed because I don't have a real time process problem.

If I have RTX4070 But I never use Mistral 7b (with awq or not). Then this is expensive garbage.

What should I do?

  • Do you guess VRAM requirements will be reduced? (like llama2.cpp, llama.c)
  • Do you guess VRAM will be increasable with RAM ([i am thinking] because Apple M3 have too much RAM than GPUs. So Nvidia release an update to increase VRAM from RAM.)
  • Have you ever use this GPU, Then share me your experiences pls.
  • What do you think about this problems?
  • Which GPU would you choose? RTX4060 TI 16GB VRAM or RTX4070 12GB VRAM?

1 Answer 1

0

Do you guess VRAM requirements will be reduced?

no. problem sizes will only become larger in the future, requiring more VRAM on the GPU if a GPU is to be used. They may refine the code (llama2.cpp, llama.c) to be more memory efficient but i doubt that will compare to the inherent memory required based on model size and model size will always become larger.

Do you guess VRAM will be increasable with RAM ([i am thinking] because Apple M3

not sure what you are asking. yes VRAM on nvidia cards will increase in the future, just like anything else and has we have noticed over the years starting from 1,2,4 gb now up to around 32gb per pcie card such as a RTX 4000 series now has. Will it coincide with DRAM DIMMS that are modular and allows someone to plug in and upgrade I do not think so. Will VRAM get up to 64gb and 128gb like we see on consumer motherboards probably. Will VRAM approach the ~768gb RAM limit of the x86-64 cpu currently, I think it is possible for a niche market but not for the consumer market.

What do you think about this problems

Large Language Models (LLM) which I know little about, a quick internet search responds with Running large language models on a home PC? - You will need at least 350GB GPU memory on your entire cluster to serve the OPT-175B model.

So it will depend on your LLM that you are playing with and what VRAM requirements you are dealing with. Playing with LLM's seems to be highly VRAM dependent, 350gb and mention of cluster gives the distinct impression that processing LLM's at home is not easily done and if so on very small models.

you also mentioned docker, which is in regards to containerization, I would look into being sure that running in a cloud instance with containers that the clustered (more than one) VRAM of many GPGPU's is actually accessible, that may be a reason why you are seeing errors such as VRAM not enough. Make sure the amount of VRAM reported within the container (docker) is correct.

Which GPU would you choose?

of the three you listed the one with the most RAM, but if that's 16gb, or even 32gb, and LLM's generally need > 100gb or whatever then recognize that reality and understand the VRAM needs for whatever LLM's you are working with. Current market price of a used Nvidia V100 gpgu having 16gb can be found around $1000 USD, they do come as 32gb and new is up to around $5000 USD, and that is with them a few generations old now (turing, ampere, current is now hopper). You would need ~4 V100's to have > 100gb of [distributed] VRAM possible along with the associated hardware to connect those. Saying low-budget and LLM in the same sentence is not realistic (yet).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.