Maniphest T3957

Clean sentry reports
Closed, MigratedEdits Locked
Actions

Assigned To

gitlab-migration

Authored By

	douardda
	Feb 18 2022, 12:14 PM

Tags

Subscribers

gitlab-migration

Description

We currently have a lot of open Sentry issues,
but this is very raw data that isn't very usable or visible.

We should either:

explicitly ignore them / stop raising them
assign them a task in the forge (manually! otherwise it would just be more noise)
fix them and/or close them

comments:

it is a process
better managing the elements that are in Sentry

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
	D7576	rDLDBASE272b5aac064e Add missing Sentry Captures

Related Objects

Mentioned In: D7583: add missing sentry captures in vault
D7582: add missing sentry captures in cvs loader
D7581: add missing sentry captures in inedexer
D7579: add missing sentry capture in deposit
D7578: Add missing sentry captures in scheduler
Mentioned Here: D2248: [WIP] In case of race condition in content_add, raise SerializationFailure instead of HashCollision.

Event Timeline

douardda triaged this task as Normal priority.Feb 18 2022, 12:14 PM

douardda created this task.

bchauvet added projects: Roadmap 2022, meta-task.Mar 23 2022, 4:57 PM

bchauvet added a revision: D7576: Add missing Sentry Captures.Apr 14 2022, 11:37 AM

bchauvet mentioned this in D7578: Add missing sentry captures in scheduler.Apr 14 2022, 1:41 PM

bchauvet mentioned this in D7579: add missing sentry capture in deposit.Apr 14 2022, 2:17 PM

bchauvet mentioned this in D7581: add missing sentry captures in inedexer.Apr 14 2022, 2:36 PM

bchauvet mentioned this in D7582: add missing sentry captures in cvs loader.Apr 14 2022, 2:40 PM

bchauvet mentioned this in D7583: add missing sentry captures in vault.Apr 14 2022, 2:46 PM

bchauvet moved this task from Backlog to Work in progress on the Roadmap 2022 board.Apr 19 2022, 10:04 AM

Sentry Sprint #1 report :

Planning

April 13th
- 10:00 - 12:00
  - Kickoff
  - Definition of the sprint goals
  - Identification of the scoped sentry issues
  - Team organization
- 14:00 - 17:00
  - First iteration
April 14th
- 10:00 - 10:30
  - Synchronization
- 10:30 - 12:00
  - Second iteration...
- 14:00 - 16:30
  - ...second iteration
- 16:30 - 17:00
  - Retrospective

Sprint Goals
- We currently have a lot of open Sentry issues, but this is very raw data that isn’t very usable or visible. They should be cleaned up so that under normal conditions, the number of reported issues stays “minimal”.
- Clean-up options:
  - Refine issues data
  - Eliminate redundant/useless issues
  - Identify sources of issues >> potential easy hacks ?

Tasks

	Task	Pair	Diffs
	turn off sentry error reporting logging integration	douardda / bchauvet	D7570
	Use explicit sentry exception capture in RPC back-end	anlambert / olasd	D7566 D7567
	RetryError Handling	anlambert / olasd	D7568
	UniqueViolation and HashCollision server-side only	vsellier / vlorentz
	Vault	ardumont/jayesh	D7566 D7564
	Define statsd metrics for sentry-removed issues	anlambert/olasd	D7569
	Add missing sentry captures	team	T3957

Pairs

vsellier - vlorentz
olasd - anlambert
ardumont - jayesh
douardda - bchauvet

Tasks details

Review the places in the code where we log exceptions to check there's an excplicit sentry capture

Decide to turn off the sentry error reporting in the logging integration https://docs.sentry.io/platforms/python/guides/logging/
- Identify and Remove expected issues:
  - - UniqueViolation (swh-storage) backend + client-side RetryError : ignore the backend issue, keep the client as the actual underlying error
    - Throttled (web app) - easy hack -> https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/exc.py$132-142
    - Not Found (silented) - easy hack -> https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/core/loader.py$349-363
  - QueryCanceled: drop the server-side error, keep the client one only
  - OSError

Define statsd metrics for sentry-removed issues
- D7569 to add an error_type tag to already existing (backend function) error metrics

UniqueViolation and HashCollision: not to be reported in sentry if the error can be encoded and sent to the client >> create a task for it

Loaders
- ValueError: Convert to NotFound exception
- OSError: to be removed
  - Pack file too big for repository xxx , limit is 4294967296 bytes, current size is 4294959115, would write 8192
- MaybeEncodingError: to be fixed (celery)
  - due to hash collision ?
- RetryError: report the underlying error instead (when the last try fails)
  - remote exception
  - hash collision
- ConnectionError: base package loader - list of known exceptions that should be logged instead of being raised to sentry
Storage
- HashCollision
  - 1CK
  - 1BW
  - -> many are false positives that are hard to resolve: https://forge.softwareheritage.org/D2248
- UniqueViolation
  - https://sentry.softwareheritage.org/organizations/swh/issues/9442/
Web App
- list of errors not sent to Sentry: https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/exc.py$132-142
- ?
- QueryCancelled: not to be logger to sentry storage-side (it's the client to report)

Vault
- NotFoundExc should not be reported to sentry https://sentry.softwareheritage.org/organizations/swh/issues/9151/

bchauvet added a commit: rDLDBASE272b5aac064e: Add missing Sentry Captures.May 2 2022, 1:37 PM

seems we still have a lot of NotFound error from the swh-loader-git:

https://sentry.softwareheritage.org/organizations/swh/issues/8800/?environment=production&project=8&query=&sort=freq&statsPeriod=7d

This task has been migrated to GitLab.