sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2025-02-17 08:04:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Make Llama 2 stop generating response on hitting specified stop words

Browse source 
It would previously some times start generating fake dialogue with
it's internal prompt patterns of <s>[INST] in responses.

This is a jarring experience. Stop generation response when hit <s>

Resolves #398
This commit is contained in:
Debanjum Singh Solanky 2023-08-01 18:53:46 -07:00
parent aa6846395d
commit 6e4050fa81
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
src/khoj/processor/conversation/gpt4all/chat_model.py
View file

@ -160,6 +160,7 @@ def llm_thread(g, messages: List[ChatMessage], model: GPT4All):
for message in conversation_history
]
stop_words = ["<s>"]
chat_history = "".join(formatted_messages)
templated_system_message = prompts.system_prompt_llamav2.format(message=system_message.content)
templated_user_message = prompts.general_conversation_llamav2.format(query=user_message.content)
@ -168,6 +169,9 @@ def llm_thread(g, messages: List[ChatMessage], model: GPT4All):
state.chat_lock.acquire()
try:
for response in response_iterator:
if any(stop_word in response.strip() for stop_word in stop_words):
logger.debug(f"Stop response as hit stop word in {response}")
break
g.send(response)
finally:
state.chat_lock.release()