Sure. Models trained using CodeGen or GPT-J from HuggingFace can be used with FauxPilot by running them through the conversion scripts in the converter directory. I also wrote a very brief set of instructions on how to fine-tune an existing model like CodeGen with HuggingFace:

It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work:

    deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json

Where run_clm.py is this script: run_clm.py

And to prepare a dataset to train on:

Have a look at the datasets library, but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like:

   {"text": "contents_of_source_file_1"}
   {"text": "contents_of_source_file_2"}
   ...

And then pass that my_code.json as the dataset name.

There are also a lot of good resources on the HuggingFace web site. As a warning, fine-tuning or training large models (like CodeGen 16B) takes a lot of GPU resources – we fine-tuned a 16B model on Verilog code, and it took 3xA100 GPUs with 80GB of VRAM each running for six days to do one pass over the 400MB dataset.

I have a question related to this (forgive my ML ignorance)

Instead of training an entirely new model, is it possible to "tweak" the existing model?

Something I have found myself wishing for a lot with Copilot is the ability to point it towards a folder containing repos of projects for a particular domain, so that it "gains knowledge" of that domain and offers better completions.

For example, when writing a database, pointing it to a folder containing the source code of Postgres, sqlite, BerkeleyDB, etc.
This way it picks up the terminology of tuple, buffer pool, scan, and the code patterns related to these things.

I know the source code to these projects is inside of Copilot's dataset, but I want the "relevance" to be weighted higher.
This could be stored as a project/workspace specific setting, like:

{
    "fauxpilot.project_additional_training_code": "/copilot/datasets/relational-databases"
}

One hack I've been using for this is to keep multiple tabs open that contain the source code to similar projects so that they are in the model's "context" (or whatever you call it).

Is there any way to accomplish something like this?

Tweaking the model on a small amount of code is what's called fine-tuning; it basically updates the model just a little bit based on a much smaller amount of data (the actual mechanics of the training process are identical to training from scratch, but fine-tuning takes much less time since it starts out much closer to the goal and doesn't have as much data to work on). Unfortunately, fine-tuning takes just as much VRAM as training from scratch, which makes it hard to do on a personal machine.

There are some architectures out there like Memorizing Transformers (https://arxiv.org/abs/2203.08913) or RETRO (https://arxiv.org/abs/2112.04426) that would be able to incorporate new information without being retrained at all, but I don't know of anyone who has made a code model that makes use of this.

@moyix I feel like most projects aren't large enough to be worth fine-tuning on. IMO it would be better to extend to context length and then put more of the project into the prompt. lucidrains says that one can replace the attention mechanism with Performer attention and fine tune — it seems like one could fine tune the 16B model on sequences of length 8192 to get "16B-long". Then a preprocessing step could be following imports and inlining relevant parts of the project into the context. That way we'd only have to finetune once