Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lookahead : add example for lookahead decoding #4207

Merged
merged 9 commits into from
Nov 26, 2023
Prev Previous commit
Next Next commit
lookahead : use deterministic init
ggerganov committed Nov 25, 2023

Verified

This commit was signed with the committer’s verified signature.
ggerganov Georgi Gerganov
commit 7bd1cd7ef48db34146491c19aa8570f994d59671
12 changes: 8 additions & 4 deletions examples/lookahead/lookahead.cpp
Original file line number Diff line number Diff line change
@@ -129,7 +129,9 @@ int main(int argc, char ** argv) {
for (int i = 0; i < W; i++) {
// initialize randomly from the prompt tokens
tokens_j[j][i] = all[1 + rand() % (all.size() - 1)];
//tokens_j[j][i] = 100 + i;

// initialize with a sequence of increasing numbers
tokens_j[j][i] = 100 + i;
}
}

@@ -327,14 +329,16 @@ int main(int argc, char ** argv) {
} else {
for (int i = 0; i < W; i++) {
// random init
tokens_j[N - 2][i] = all[1 + rand() % (all.size() - 1)];
//tokens_j[N - 2][i] = tokens_j[0][i];
//tokens_j[N - 2][i] = all[1 + rand() % (all.size() - 1)];

// init from the previous level
tokens_j[N - 2][i] = tokens_j[0][i];
}
}
}

// update observed ngrams
{
if (v == 0) {
// the first token of the n-gram is determined by the index in the container so it is not stored
std::vector<llama_token> ngram(N - 1);