Trying self hosting alpaca a finetuned llama model after OpenAI drama
With all the OpenAI drama, time for me to try self hosting Alpaca, a finetuning ChatGPT-type model of LLaMA, which is Facebook's leaked GPT model
Now I finally get the origins of ChatGPT, it's just instruct models, same as instruct-pix2pix we had in Stable Diffusion, e.g. "change the haircolor to red" but for text
And instruct-text is what OpenAI smartly branded as "ChatGPT" and that became huge
Fun to see the technical origins of something so big were just available to us all
The finetuning is way way way way easier than I thought, not some magic but just a JSON text file with questions and answers which teachers the LLM model to reply to questions with answers, e.g. a chat
It's exactly the same what I use for Photo AI etc. finetuning with Dreambooth or LoRa, it's almost exactly the same tech, but you finetune on photos of people instead of text files, obviously I knew that but interesting to actually see it in front of you
https://github.com/tatsu-lab/stanford_alpaca
P.S. I'm on Twitter too if you'd like to follow more of my stories. And I wrote a book called MAKE about building startups without funding. See a list of my stories or contact me. To get an alert when I write a new blog post, you can subscribe below:
