Hi,
I was reading the Byte Pair Encoding (BPE) example on Page 91 of the book, and I have a question regarding the frequency calculation in Iteration 5.
Specifically, the frequency of the merged token “s” with is listed as 8, but the frequency of “s” is only 7 prior to the merge. Need some clarification on this
• How can “s” appear 8 times when “s” only occurs 7 times in the corpus?
Could you please clarify how this frequency was computed? Is there an additional detail about how token pairs are counted that might not be immediately apparent from the example?
Thank you for your time!
Raghu
Hi,
I was reading the Byte Pair Encoding (BPE) example on Page 91 of the book, and I have a question regarding the frequency calculation in Iteration 5.
Specifically, the frequency of the merged token “s” with is listed as 8, but the frequency of “s” is only 7 prior to the merge. Need some clarification on this
• How can “s” appear 8 times when “s” only occurs 7 times in the corpus?
Could you please clarify how this frequency was computed? Is there an additional detail about how token pairs are counted that might not be immediately apparent from the example?
Thank you for your time!
Raghu