rjmalagon / dolphin-2.9.3-mistral

工具 7B

31 拉取 3周前更新

3周前更新

3周前

495ffc5e6977 · 7.7GB

{"stop":["[INST]","[/INST]"]}

30B

模板

{{- if .Messages }} {{- range $index, $_ := .Messages }} {{- if eq .Role "user" }} {{- if and (eq (len (slice $.Messages $index)) 1) $.Tools }}[AVAILABLE_TOOLS] {{ $.Tools }}[/AVAILABLE_TOOLS] {{- end }}[INST] {{ if and $.System (eq (len (slice $.Messages $index)) 1) }}{{ $.System }} {{ end }}{{ .Content }}[/INST] {{- else if eq .Role "assistant" }} {{- if .Content }} {{ .Content }} {{- else if .ToolCalls }}[TOOL_CALLS] [ {{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{- end }}] {{- end }}</s> {{- else if eq .Role "tool" }}[TOOL_RESULTS] {"content": {{ .Content }}} [/TOOL_RESULTS] {{- end }} {{- end }} {{- else }}[INST] {{ if .System }}{{ .System }} {{ end }}{{ .Prompt }}[/INST] {{- end }} {{ .Response }} {{- if .Response }}</s> {{- end }}

801B

说明文档

Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

由Eric Hartford和Cognitive Computations整理和训练

Discord: https://discord.gg/h3K4XGj2RH

感谢Dolphin 2.9.3的赞助者
- Crusoe Cloud - 提供了优质的8xH100节点
- OnDemand - 提供了推理赞助

此模型基于mistralai/Mistral-7B-v0.3，受Apache 2.0许可证管理。

基础模型具有32k的上下文，我们的微调使用了8192序列长度。

Dolphin 2.9.3使用ChatML提示模板格式。

示例

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Dolphin-2.9.3具有各种指令遵循、对话和编码技能。它还具有初始的代理能力并支持函数调用。

Dolphin未经审查。我们已经过滤了数据集以移除一致性和偏差。这使得模型更符合规定。建议在将模型作为服务公开之前实现自己的对齐层。它将对任何请求都高度合规，包括不道德的请求。请阅读关于未审查模型的博客文章。https://erichartford.com/uncensored-models 对使用此模型创建的任何内容负责。请负责任地享受。

Dolphin根据Apache 2.0许可证进行授权。我们授予任何用途的许可，包括商业用途。Dolphin在GPT4等模型生成数据上进行了训练。

评估

https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k

培训

参阅axolotl配置

axolotl 版本： 0.4.0
”`yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false

load_in_4bit: true

strict: false

datasets
- path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
type: sharegpt
conversation: chatml
- path: /workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
type: sharegpt
conversation: chatml

chat_template: chatml

adapter: qlora

lora_r: 128

lora_alpha: 16

lora_modules_to_save: [embed_tokens, lm_head]

lora_dropout: 0.05

lora_target_linear: true

dataset_prepared_path: /workspace/axolotl/dolph-2.9.3-prepared
val_set_size: 0.01
output_dir: /workspace/axolotl/dolphin-2.9.3-mistral-7B

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: dolphin-2.9.3-Mistral-7B
wandb_watch
wandb_run_id
wandb_log_model

gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 5e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16
tf32

gradient_checkpointing: true
gradient_checkpointing_kwargs
use_reentrant: false
early_stopping_patience
resume_from_checkpoint
logging_steps: 1
xformers_attention
flash_attention: true

warmup_steps: 100

evals_per_epoch: 4

eval_table_size
saves_per_epoch: 1
save_total_limit: 2
save_steps
debug
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.1
fsdp
fsdp_config
special_tokens
eos_token: “<|im_end|>”
tokens
- “<|im_start|>”

# Dolphin 2.9.3 Mistral 7b v0.3 32k 🐬

Curated and trained by Eric Hartford and Cognitive Computations

[![Discord](https://img.shields.io/discord/1156064224225808488?logo=Discord&logoColor=%23ffffff&label=Discord&link=https%3A%2F%2Fdiscord.gg%2FtCMkMDDHwm)](https://discord.gg/h3K4XGj2RH)
Discord: https://discord.gg/h3K4XGj2RH

Our appreciation for the sponsors of Dolphin 2.9.3:
- [Crusoe Cloud](https://crusoe.ai/) - provided excellent on-demand 8xH100 node
- [OnDemand](https://on-demand.io/) - provided inference sponsorship

This model is based on mistralai/Mistral-7B-v0.3, and is governed by the apache 2.0 license.

The base model has 32k context, and our finetuning took place with 8192 sequence length.

Dolphin 2.9.3 uses ChatML prompt template format.

example:

```
<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

```

Dolphin-2.9.3 has a variety of instruction following, conversational, and coding skills. It also has initial agentic abilities and supports function calling.

Dolphin is uncensored. We have filtered the dataset to remove alignment and bias. This makes the model more compliant. You are advised to implement your own alignment layer before exposing the model as a service. It will be highly compliant with any requests, even unethical ones. Please read my blog post about uncensored models. https://erichartford.com/uncensored-models You are responsible for any content you create using this model. Enjoy responsibly.

Dolphin is licensed according to apache 2.0 license.  We grant permission for any use, including commercial. Dolphin was trained on data generated from GPT4, among other models.

## Evals

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/5KUgfzJyY1IM4Yg6bg3Dq.png)

[https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k](https://hugging-face.cn/cognitivecomputations/dolphin-2.9.3-mistral-7B-32k)
## Training

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.0`
```yaml
base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
# load_in_4bit: true
strict: false

datasets:
  - path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
    type: sharegpt  
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
    type: sharegpt
    conversation: chatml
  - path: /workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
    type: sharegpt 
    conversation: chatml

chat_template: chatml
# adapter: qlora
# lora_r: 128
# lora_alpha: 16
# lora_modules_to_save: [embed_tokens, lm_head]
# lora_dropout: 0.05
# lora_target_linear: true

dataset_prepared_path:  /workspace/axolotl/dolph-2.9.3-prepared
val_set_size: 0.01
output_dir: /workspace/axolotl/dolphin-2.9.3-mistral-7B

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: dolphin-2.9.3-Mistral-7B
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 5e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32:

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
# evals_per_epoch: 4
eval_table_size:
saves_per_epoch: 1
save_total_limit: 2
save_steps:
debug:
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
  eos_token: "<|im_end|>"
tokens:
  - "<|im_start|>"

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)