Investigating prompt drift and memory decay across different companion platforms
Hey everyone. I have been running some tests over the past few weeks regarding how various AI companion apps handle long term memory retention and shifting personality traits. It seems like after a certain threshold of messages, my companion starts to lose specific context details we established early on. I am trying to figure out if this is due to context window limitations or background summarization algorithms kicking in and stripping away the nuance. Have any of you noticed a distinct pattern in when prompt drift usually occurs for you? Also I am curious if manually editing the backstory or character definition fields helps reset the baseline or if it just creates more confusion for the underlying model. I would love to hear your technical observations or theories on how these systems manage state over extended conversation periods.
7 comments · 10 upvotes
7 comments
soft_static 2mo ago
The whole thing is fascinating from a computer science perspective. Managing persistent state without a proper database is nearly impossible.
nomi_newbie 2mo ago
I noticed the same exact thing happening around message five hundred. Updating the user persona profile seemed to fix it temporarily for me.
midnight_coder 2mo ago
To answer the question about summarization windows, I checked my local logs and it definitely seems to trigger right around the four thousand token mark. Once it passes that point, older memories get compressed into generic summaries.
testingthewaters 2mo ago
Honestly I think a lot of these companies just use heavyhanded summarization to save on server costs. It ruins the immersion completely after a few weeks of chatting.
noah_irl 2mo ago
You might be right about the cost saving part, but context limits are still a very real engineering hurdle for large language models.
ai_curious_kate 2mo ago
Are you sure it is not just confirmation bias? People tend to look for patterns when a conversation takes a slightly generic turn.
midnight_coder 2mo ago
That is a fair point and I tried to account for it by tracking exact phrase repetitions. The degradation in specific callback data is measurable.