rjmalagon / dolphin-2.9.3-mistral

工具 7B

31 拉取 3周前更新

3周前更新

3周前

1f9baf050efa · 14GB

<|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant

105B

params

{"stop":["<|im_end|>","<|im_start|>"]}

59B

Readme

Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

由Eric Hartford和Cognitive Computations精心培养和训练

Discord: https://discord.gg/h3K4XGj2RH

感谢Dolphin 2.9.3的赞助商
- Crusoe Cloud - 提供了优秀的按需8xH100节点
- OnDemand - 提供了推理赞助

本模型基于mistralai/Mistral-7B-v0.3，并受Apache 2.0许可的约束。

基本模型有32k上下文，我们的微调是在8192序列长度下进行的。

Dolphin 2.9.3使用ChatML提示模板格式。

示例

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Dolphin-2.9.3具有多种指令遵循、对话和编码技能。它还具有初始的代理能力并支持函数调用。

Dolphin未审查。我们已过滤数据集以删除对齐和偏差。这使模型更合规。请在将模型作为服务公开之前实施自己的对齐层。它将高度遵守任何请求，即使是不道德的请求。请阅读我的关于未审查模型的博客文章。[链接](https://erichartford.com/uncensored-models) 你对本模型创建的任何内容负责。请负责任地享受。

Dolphin根据Apache 2.0许可证授权。我们允许任何使用，包括商业使用。Dolphin在GPT4等模型生成的数据上进行了训练。

评估

https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k

训练

参见axolotl配置

axolotl版本：0.4.0
”`yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false

load_in_4bit: true

strict: false

datasets
- path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
type: sharegpt
conversation: chatml
路径：/workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
type: sharegpt
conversation: chatml

聊天模板：chatml

适配器：qlora

lora_r：128

lora_alpha：16

lora_modules_to_save：[embed_tokens, lm_head]

lora_dropout：0.05

lora_target_linear：true

数据集准备路径：/workspace/axolotl/dolph-2.9.3-prepared
验证集大小：0.01
输出目录：/workspace/axolotl/dolphin-2.9.3-mistral-7B

序列长度：8192
样本打包：true
填充到序列长度：true

wandb项目：dolphin-2.9.3-Mistral-7B
wandb观察
wandb运行ID
wandb记录模型

梯度累积步骤：16
微批次大小：1
训练轮数：3
优化器：adamw_8bit
学习率计划器：cosine
学习率：5e-6
在输入上训练：false
按长度分组：false
bf16：auto
fp16
tf32

梯度检查：true
梯度检查关键字参数
使用重入：false
早停耐心值
从检查点恢复
记录步骤：1
xformers注意力
flash注意力：true

预热步骤：100

每轮评估次数：4

评估表大小
每轮保存次数：1
保存总限制：2
保存步骤
调试
deepspeed：/workspace/axolotl/deepspeed_configs/zero3_bf16.json
权重衰减：0.1
fsdp
fsdp配置
特殊令牌
eos令牌：“<|im_end|>”
令牌
- “<|im_start|>”

# Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

Curated and trained by Eric Hartford and Cognitive Computations

[![Discord](https://img.shields.io/discord/1156064224225808488?logo=Discord&logoColor=%23ffffff&label=Discord&link=https%3A%2F%2Fdiscord.gg%2FtCMkMDDHwm)](https://discord.gg/h3K4XGj2RH)
Discord: https://discord.gg/h3K4XGj2RH

Our appreciation for the sponsors of Dolphin 2.9.3:
- [Crusoe Cloud](https://crusoe.ai/) - provided excellent on-demand 8xH100 node
- [OnDemand](https://on-demand.io/) - provided inference sponsorship

This model is based on mistralai/Mistral-7B-v0.3, and is governed by the apache 2.0 license.

The base model has 32k context, and our finetuning took place with 8192 sequence length.

Dolphin 2.9.3 uses ChatML prompt template format.

example:

```
<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

```

Dolphin-2.9.3 has a variety of instruction following, conversational, and coding skills. It also has initial agentic abilities and supports function calling.

Dolphin is uncensored. We have filtered the dataset to remove alignment and bias. This makes the model more compliant. You are advised to implement your own alignment layer before exposing the model as a service. It will be highly compliant with any requests, even unethical ones. Please read my blog post about uncensored models. https://erichartford.com/uncensored-models You are responsible for any content you create using this model. Enjoy responsibly.

Dolphin is licensed according to apache 2.0 license.  We grant permission for any use, including commercial. Dolphin was trained on data generated from GPT4, among other models.

## Evals

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/5KUgfzJyY1IM4Yg6bg3Dq.png)

[https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k](https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k)
## Training

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.0`
```yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
# load_in_4bit: true
strict: false

datasets:
  - path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
    type: sharegpt 
    conversation: chatml

chat_template: chatml
# adapter: qlora
# lora_r: 128
# lora_alpha: 16
# lora_modules_to_save: [embed_tokens, lm_head]
# lora_dropout: 0.05
# lora_target_linear: true

dataset_prepared_path:  /workspace/axolotl/dolph-2.9.3-prepared
val_set_size: 0.01
output_dir: /workspace/axolotl/dolphin-2.9.3-mistral-7B

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: dolphin-2.9.3-Mistral-7B
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 5e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32:

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
# evals_per_epoch: 4
eval_table_size:
saves_per_epoch: 1
save_total_limit: 2
save_steps:
debug:
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
  eos_token: "<|im_end|>"
tokens:
  - "<|im_start|>"

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)