rjmalagon / dolphin-2.9.3-mistral

工具 7B

31 Pulls 更新于3周前

更新于3周前

3周前

1f9baf050efa · 14GB

<|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant

105B

参数

{"stop":["<|im_end|>","<|im_start|>"]}

59B

说明文档

Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

由 Eric Hartford 和 Cognitive Computations 精选和训练

Discord: https://discord.gg/h3K4XGj2RH

感谢赞助 Dolphin 2.9.3 的赞助商
- Crusoe Cloud - 提供了优质的 8xH100 节点
- OnDemand - 提供了推理赞助

本模型基于 mistralai/Mistral-7B-v0.3，遵守 Apache 2.0 许可协议。

基本模型具有 32k 上下文，我们的微调使用了 8192 序列长度。

Dolphin 2.9.3 使用 ChatML 提示模板格式。

示例

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Dolphin-2.9.3 具有多种指令遵循、对话和编码技能。它还具备初始的代理能力，并支持函数调用。

Dolphin 没有禁止内容。我们过滤了数据集以消除一致性和偏见。这使得模型更加合规。在将模型作为服务公开之前，建议您实现自己的对齐层。它将对任何请求都高度合规，甚至包括不道德的请求。请阅读我的关于未审查模型的博客文章。 https://erichartford.com/uncensored-models 您对使用本模型创建的任何内容负有责任。请负责任地享受。

Dolphin 根据 Apache 2.0 许可协议授权。我们允许任何用途，包括商业用途。Dolphin 在 GPT4 等其他模型生成数据的基础上进行了训练。

评估

https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k

训练

查看 axolotl 配置

axolotl 版本: 0.4.0
”`yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false

load_in_4bit: true

strict: false

datasets
- path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
type: sharegpt
conversation: chatml
- 路径：/workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
type: sharegpt
conversation: chatml

chat_template: chatml

适配器：qlora

lora_r: 128

lora_alpha: 16

lora_modules_to_save: [embed_tokens, lm_head]

lora_dropout: 0.05

lora_target_linear: true

数据集准备路径：/workspace/axolotl/dolph-2.9.3-prepared
验证集大小：0.01
输出目录：/workspace/axolotl/dolphin-2.9.3-mistral-7B

序列长度：8192
样本打包：true
填充到序列长度：true

wandb_project: dolphin-2.9.3-Mistral-7B
wandb_watch
wandb_run_id
wandb_log_model

梯度累积步骤：16
微观批大小：1
epoch数量：3
优化器：adamw_8bit
学习率调度器：cosine
学习率：5e-6
在输入上训练：false
按长度分组：false
bf16：auto
fp16
tf32

梯度 checkpointing：true
梯度 checkpointing_kwargs
use_reentrant：false
早期停止耐心度：early_stopping_patience
从中断处恢复：resume_from_checkpoint
日志步骤：1
xformers_attention
flash_attention：true

预加热步骤：100

每个epoch评估次数：4

评估表大小：eval_table_size
每个epoch保存次数：1
保存总限制：2
保存步骤：save_steps
调试：debug
deepspeed：/workspace/axolotl/deepspeed_configs/zero3_bf16.json
权重衰减：0.1
fsdp
fsdp_config
特殊令牌
eos_token: “<|im_end|>”
tokens
- “<|im_start|>”

# Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

Curated and trained by Eric Hartford and Cognitive Computations

[![Discord](https://img.shields.io/discord/1156064224225808488?logo=Discord&logoColor=%23ffffff&label=Discord&link=https%3A%2F%2Fdiscord.gg%2FtCMkMDDHwm)](https://discord.gg/h3K4XGj2RH)
Discord: https://discord.gg/h3K4XGj2RH

Our appreciation for the sponsors of Dolphin 2.9.3:
- [Crusoe Cloud](https://crusoe.ai/) - provided excellent on-demand 8xH100 node
- [OnDemand](https://on-demand.io/) - provided inference sponsorship

This model is based on mistralai/Mistral-7B-v0.3, and is governed by the apache 2.0 license.

The base model has 32k context, and our finetuning took place with 8192 sequence length.

Dolphin 2.9.3 uses ChatML prompt template format.

example:

```
<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

```

Dolphin-2.9.3 has a variety of instruction following, conversational, and coding skills. It also has initial agentic abilities and supports function calling.

Dolphin is uncensored. We have filtered the dataset to remove alignment and bias. This makes the model more compliant. You are advised to implement your own alignment layer before exposing the model as a service. It will be highly compliant with any requests, even unethical ones. Please read my blog post about uncensored models. https://erichartford.com/uncensored-models You are responsible for any content you create using this model. Enjoy responsibly.

Dolphin is licensed according to apache 2.0 license.  We grant permission for any use, including commercial. Dolphin was trained on data generated from GPT4, among other models.

## Evals

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/5KUgfzJyY1IM4Yg6bg3Dq.png)

[https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k](https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k)
## Training

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.0`
```yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
# load_in_4bit: true
strict: false

datasets:
  - path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
    type: sharegpt 
    conversation: chatml

chat_template: chatml
# adapter: qlora
# lora_r: 128
# lora_alpha: 16
# lora_modules_to_save: [embed_tokens, lm_head]
# lora_dropout: 0.05
# lora_target_linear: true

dataset_prepared_path:  /workspace/axolotl/dolph-2.9.3-prepared
val_set_size: 0.01
output_dir: /workspace/axolotl/dolphin-2.9.3-mistral-7B

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: dolphin-2.9.3-Mistral-7B
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 5e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32:

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
# evals_per_epoch: 4
eval_table_size:
saves_per_epoch: 1
save_total_limit: 2
save_steps:
debug:
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
  eos_token: "<|im_end|>"
tokens:
  - "<|im_start|>"

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)