N-gram 相似度计算与评估

示例文本

• 参考文本: “Natural language processing is very interesting.”
• 生成文本: “Natural language processing is quite interesting.”

1. 提取 N-gram

1.1. Unigram（1-gram）

参考文本的Unigram:

["Natural", "language", "processing", "is", "very", "interesting"] 

生成文本的Unigram:

["Natural", "language", "processing", "is", "quite", "interesting"] 

1.2. Bigram（2-gram）

参考文本的Bigram:

["Natural language", "language processing", "processing is", "is very", "very interesting"] 

生成文本的Bigram:

["Natural language", "language processing", "processing is", "is quite", "quite interesting"] 

2. 计算 N-gram 重叠

2.1. Unigram 重叠

重叠的Unigram:

["Natural", "language", "processing", "is", "interesting"] 

重叠的Unigram数量：5

总参考文本Unigram数量：6

总生成文本Unigram数量：6

2.2. Bigram 重叠

重叠的Bigram:

["Natural language", "language processing", "processing is"] 

重叠的Bigram数量：3

总参考文本Bigram数量：5

总生成文本Bigram数量：5

3. 计算评估指标

3.1. 精确率（Precision）

• Unigram 精确率：
  $${\text{Precision}}_{\text{Unigram}}=\frac{\text{重叠的Unigram数量}}{\text{生成文本中的总Unigram数量}}=\frac{5}{6}\approx 0.83$$

• Bigram 精确率：
  $${\text{Precision}}_{\text{Bigram}}=\frac{\text{重叠的Bigram数量}}{\text{生成文本中的总Bigram数量}}=\frac{3}{5}=0.6$$

3.2. 召回率（Recall）

• Unigram 召回率：
  $${\text{Recall}}_{\text{Unigram}}=\frac{\text{重叠的Unigram数量}}{\text{参考文本中的总Unigram数量}}=\frac{5}{6}\approx 0.83$$

• Bigram 召回率：
  $${\text{Recall}}_{\text{Bigram}}=\frac{\text{重叠的Bigram数量}}{\text{参考文本中的总Bigram数量}}=\frac{3}{5}=0.6$$

3.3. F1 分数（F1 Score）

• Unigram F1分数：
  $${\text{F1}}_{\text{Unigram}}=2\times \frac{{\text{Precision}}_{\text{Unigram}}\times {\text{Recall}}_{\text{Unigram}}}{{\text{Precision}}_{\text{Unigram}}+{\text{Recall}}_{\text{Unigram}}}=2\times \frac{0.83\times 0.83}{0.83+0.83}\approx 0.83$$

• Bigram F1分数：
  $${\text{F1}}_{\text{Bigram}}=2\times \frac{{\text{Precision}}_{\text{Bigram}}\times {\text{Recall}}_{\text{Bigram}}}{{\text{Precision}}_{\text{Bigram}}+{\text{Recall}}_{\text{Bigram}}}=2\times \frac{0.6\times 0.6}{0.6+0.6}=0.6$$

4. Python 代码实现

from collections import Counter from typing import List  def extract_ngrams(text: str, n: int) -> List[str]:     words = text.split()     return [' '.join(words[i:i+n]) for i in range(len(words)-n+1)]  def calculate_ngram_overlap(reference: str, generated: str, n: int):     reference_ngrams = extract_ngrams(reference, n)     generated_ngrams = extract_ngrams(generated, n)          reference_counter = Counter(reference_ngrams)     generated_counter = Counter(generated_ngrams)          overlapping_ngrams = set(reference_counter.keys()) & set(generated_counter.keys())          overlap_count = sum(min(reference_counter[ngram], generated_counter[ngram]) for ngram in overlapping_ngrams)     precision = overlap_count / len(generated_ngrams) if generated_ngrams else 0     recall = overlap_count / len(reference_ngrams) if reference_ngrams else 0     f1_score = 2 * (precision * recall) / (precision + recall) if (precision + recall) else 0          return precision, recall, f1_score  reference_text = "Natural language processing is very interesting." generated_text = "Natural language processing is quite interesting."  # 计算Unigram（1-gram） precision_unigram, recall_unigram, f1_unigram = calculate_ngram_overlap(reference_text, generated_text, 1) print(f"Unigram - 精确率: {precision_unigram:.2f}, 召回率: {recall_unigram:.2f}, F1分数: {f1_unigram:.2f}")  # 计算Bigram（2-gram） precision_bigram, recall_bigram, f1_bigram = calculate_ngram_overlap(reference_text, generated_text, 2) print(f"Bigram - 精确率: {precision_bigram:.2f}, 召回率: {recall_bigram:.2f}, F1分数: {f1_bigram:.2f}") 

Code

完整示例代码已上传至：Machine Learning and Deep Learning Algorithms with NumPy
此项目包含更多AI相关的算法numpy实现，供大家学习参考使用，欢迎star~

备注

个人水平有限，有问题随时交流~