chore(grouping): Include grouphash age when logging parent_hash_missing_group events
#88635
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅ ✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## master #88635 +/- ##
=======================================
Coverage 87.72% 87.72%
=======================================
Files 10064 10064
Lines 569157 569168 +11
Branches 22351 22351
=======================================
+ Hits 499269 499293 +24
+ Misses 69508 69495 -13
Partials 380 380 |
0499a9e to
a32f1a0
Compare
wedamija
approved these changes
Apr 3, 2025
a32f1a0 to
5d55917
Compare
JoshFerge
reviewed
Apr 4, 2025
JoshFerge
approved these changes
Apr 4, 2025
andrewshie-sentry
pushed a commit
that referenced
this pull request
Apr 8, 2025
…ing_group` events (#88635) Sometimes, Seer will identify as a match a grouphash which doesn't have a group. In theory this shouldn't happen (all of the data in Seer theoretically comes from events with groups, and when a group is deleted, we trigger a delete in Seer, too), but we consistently see a few dozen such errors an hour. Our best guess to explain this is a race condition: Event A gets sent to Seer, doesn't find a match, and gets put in the Seer database, and then before it's been able to create a group for itself, event B gets sent to Seer and matches with event A. One of the (multiple) initial motivations for storing grouphash metadata was to test this theory out. Since I've recently come back to our Seer ingest code, I finally got around to doing so, by adding the age in seconds of the group-less grouphash to the log we collect when we run into this. This should let us distinguish race-condition-caused instances (where the age should be under a second) from ones caused by something else (where the age could be anything). Once we have a better sense of what's going on, we can decide whether we want to exclude the race condition cases from the record deletion that hitting this situation triggers in Seer.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometimes, Seer will identify as a match a grouphash which doesn't have a group. In theory this shouldn't happen (all of the data in Seer theoretically comes from events with groups, and when a group is deleted, we trigger a delete in Seer, too), but we consistently see a few dozen such errors an hour.
Our best guess to explain this is a race condition: Event A gets sent to Seer, doesn't find a match, and gets put in the Seer database, and then before it's been able to create a group for itself, event B gets sent to Seer and matches with event A. One of the (multiple) initial motivations for storing grouphash metadata was to test this theory out. Since I've recently come back to our Seer ingest code, I finally got around to doing so, by adding the age in seconds of the group-less grouphash to the log we collect when we run into this. This should let us distinguish race-condition-caused instances (where the age should be under a second) from ones caused by something else (where the age could be anything).
Once we have a better sense of what's going on, we can decide whether we want to exclude the race condition cases from the record deletion that hitting this situation triggers in Seer.