jcai / breeze-7b-32k-instruct-v1_0

Breeze-7B-32k-Instruct是从基础模型Breeze-7B-32k-Base派生的，使得得到的模型可以用于常见的任务。〈f16, Q4, Q4_K_M〉

158 pulls 两个月前更新

两个月前更新

两个月前

1a2e1735c224 · 15GB

README

MediaTek Research Breeze-7B-32k-Instruct-v1_0的模型卡片

来源：链接

MediaTek Research Breeze-7B（以下简称 Breeze-7B）是一个基于Mistral-7B的语言模型系列，专为繁体中文使用而设计。

Breeze-7B-Base是 Breeze-7B 系列的基础模型。
如果您有大量的微调数据，此模型适用于为特定用例微调。

Breeze-7B-Instruct 从基础模型 Breeze-7B-Base 衍生而来，使得结果模型适用于常见的任务。

Breeze-7B-32k-Base 在基础模型的基础上扩展了更多数据、基础变化和禁用滑动窗口。
大致相当于44k个繁体中文字符。

Breeze-7B-32k-Instruct 从基础模型 Breeze-7B-32k-Base 衍生而来，使得结果模型适用于常见的任务。

实用性方面
— Breeze-7B-Base 通过增加额外的 30,000 个繁体中文标记扩展了原始词汇表。在所有其他条件相同的情况下，Breeze-7B 在处理繁体中文时，相比于 Mistral-7B 和 Llama 7B，推理速度提高了两倍。[参见推理性能。]
— Breeze-7B-Instruct 可用于常见的问答、RAG、多轮对话和摘要等任务。
— Breeze-7B-32k-Instruct 可以在文档级别执行任务（对于中文，20 ~ 40 页）。

该项目的成员包括（按字母顺序排序）：許湛然、廖峰挺、許博竣、陳宜昌，以及指导老师許大山。

功能

Breeze-7B-32k-Base-v1_0
- 将词汇表字典大小从 32k 扩展到 62k，以更好地支持繁体中文
- 32k tokens 的上下文长度
Breeze-7B-32k-Instruct-v1_0
- 将词汇表字典大小从 32k 扩展到 62k，以更好地支持繁体中文
- 32k tokens 的上下文长度
- 多轮对话（无需对有害性进行特殊处理）

模型详情

Breeze-7B-32k-Base-v1_0
- 预训练自：Breeze-7B-Base
- 模型类型：因果解耦变压器语言模型
- 语言：英语和繁体中文（zh-tw）
Breeze-7B-32k-Instruct-v1_0
- 微调自：Breeze-7B-32k-Base
- 模型类型：因果解耦变压器语言模型
- 语言：英语和繁体中文（zh-tw）

长上下文性能

海中拾针性能

我们通过关键检索任务来测试模型在给定序列中关注不同深度的能力。
在长上下文中放置了要检索的关键，以便于模型检索。
关键位置被归入 16 个区间，每个区间有 20 个测试案例。
Breeze-7B-32k-Base 以 90% 以上的准确率完成这些任务，如图所示。

长 DRCD 性能

模型/性能(EM) / DRCD	DRCD	DRCD-16k	DRCD-32k
Breeze-7B-32k-Instruct-v1_0	76.9	54.82	44.26
Breeze-7B-32k-Base-v1_0	79.73	69.68	61.55
Breeze-7B-Base-v1_0	80.61	21.79	15.29

短期基准性能

模型/性能(EM) / DRCD	TMMLU+	MMLU	TABLE	MT-Bench-tw	MT-Bench
Breeze-7B-32k-Instruct-v1_0	41.37	61.34	34	5.8	7.4
Breeze-7B-Instruct-v1_0	42.67	62.73	39.58	6.0	7.4

引用

@article{MediaTek-Research2024breeze7b,
      title={Breeze-7B Technical Report}, 
      author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},
      year={2024},
      eprint={2403.02712},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

## Model Card for MediaTek Research Breeze-7B-32k-Instruct-v1_0

## Source: [Link](https://hugging-face.cn/MediaTek-Research/Breeze-7B-32k-Instruct-v1_0)

MediaTek Research Breeze-7B (hereinafter referred to as Breeze-7B) is a language model family that builds on top of [Mistral-7B](https://hugging-face.cn/mistralai/Mistral-7B-v0.1), specifically intended for Traditional Chinese use.

[Breeze-7B-Base](https://hugging-face.cn/MediaTek-Research/Breeze-7B-Base-v1_0) is the base model for the Breeze-7B series. 
It is suitable for use if you have substantial fine-tuning data to tune it for your specific use case.

[Breeze-7B-Instruct](https://hugging-face.cn/MediaTek-Research/Breeze-7B-Instruct-v1_0) derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks.

[Breeze-7B-32k-Base](https://hugging-face.cn/MediaTek-Research/Breeze-7B-32k-Base-v1_0) is extended from the base model with more data, base change, and the disabling of the sliding window. 
Roughly speaking, that is equivalent to 44k Traditional Chinese characters.

[Breeze-7B-32k-Instruct](https://hugging-face.cn/MediaTek-Research/Breeze-7B-32k-Instruct-v1_0) derives from the base model Breeze-7B-32k-Base, making the resulting model amenable to be used as-is for commonly seen tasks.

Practicality-wise:
- Breeze-7B-Base expands the original vocabulary with additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, everything else being equal, Breeze-7B operates at twice the inference speed for Traditional Chinese to Mistral-7B and Llama 7B. [See [Inference Performance](#inference-performance).]
- Breeze-7B-Instruct can be used as is for common tasks such as Q&A, RAG, multi-round chat, and summarization.
- Breeze-7B-32k-Instruct can perform tasks at a document level (For Chinese, 20 ~ 40 pages).

*A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*

## Features

- Breeze-7B-32k-Base-v1_0
  - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
  - 32k-token context length
 
- Breeze-7B-32k-Instruct-v1_0
  - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
  - 32k-token context length
  - Multi-turn dialogue (without special handling for harmfulness)

## Model Details

- Breeze-7B-32k-Base-v1_0
  - Pretrained from: [Breeze-7B-Base](https://hugging-face.cn/MediaTek-Research/Breeze-7B-Base-v1_0)
  - Model type: Causal decoder-only transformer language model
  - Language: English and Traditional Chinese (zh-tw)
- Breeze-7B-32k-Instruct-v1_0
  - Finetuned from: [Breeze-7B-32k-Base](https://hugging-face.cn/MediaTek-Research/Breeze-7B-32k-Base-v1_0)
  - Model type: Causal decoder-only transformer language model
  - Language: English and Traditional Chinese (zh-tw)

## Long-context Performance

#### Needle-in-a-haystack Performance

We use the passkey retrieval task to test the model's ability to attend to different various depths in a given sequence.
A key in placed within a long context distracting document for the model to retrieve.
The key position is binned into 16 bins, and there are 20 testcases for each bin.
Breeze-7B-32k-Base clears the tasks with 90+% accuracy, shown in the figure below.
![Needle-in-a-haystack Performance](https://hugging-face.cn/MediaTek-Research/Breeze-7B-32k-Base-v1_0/resolve/main/needle-in-a-haystack-performance.png)

#### Long-DRCD Performance

| **Model/Performance(EM)** | **DRCD** | **DRCD-16k** | **DRCD-32k** |
|---------------------------|----------|--------------|--------------|
| **Breeze-7B-32k-Instruct-v1\_0** |  76.9   |   54.82   |  44.26     |       
| **Breeze-7B-32k-Base-v1\_0** | 79.73    | 69.68        | 61.55        |
| **Breeze-7B-Base-v1\_0**      | 80.61    | 21.79        | 15.29        |

#### Short-Benchmark Performance

| **Model/Performance(EM)** | **TMMLU+** | **MMLU** | **TABLE** | **MT-Bench-tw** | **MT-Bench** |
|---------------------------|----------|--------------|--------------|-----|-----|
| **Breeze-7B-32k-Instruct-v1\_0** | 41.37    |   61.34   |   34    | 5.8 | 7.4 |
| **Breeze-7B-Instruct-v1\_0**      |  42.67  |   62.73    |   39.58 | 6.0 | 7.4 |

## Citation

```
@article{MediaTek-Research2024breeze7b,
      title={Breeze-7B Technical Report}, 
      author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},
      year={2024},
      eprint={2403.02712},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)