[FEATURE] Automate Rule-Based Anomaly Detection with Feedback-Driven Threshold Adjustment

[kaituo]

Is your feature request related to a problem?
Develop a system that continuously adjusts anomaly detection thresholds based on user feedback. The objective is to automatically infer optimal thresholds—using both historical and real-time analysis—to reduce false positives and enhance detection accuracy.

What solution would you like?

• Feedback Integration: Seamlessly incorporate user feedback into the anomaly detection workflow.
• Dynamic Threshold Adjustment: Automatically update thresholds using historical and real-time data.
• User-Centric Experience: Provide an intuitive UX for submitting feedback and reviewing suggested rule changes.

What alternatives have you considered?

Do you have any additional context?

• Rule-based AD Blog:
  Rule-based AD Blog Post
• RFC Proposal:
  RFC - Proposal: Semi-supervised Anomaly Detection

Milestones and Deliverables

1. UX for Feedback Collection

Objective:
Design and implement an interactive interface to capture and process user feedback.

Key Tasks:

• AD frontend
  • Create a flyout/modal that prompts users to confirm anomaly status changes (e.g., marking an anomaly as normal).
  • Display before/after views to clearly communicate the impact of the feedback.

2. Backend Integration for Feedback Propagation

Objective:
channels user feedback to backend.

Key Tasks:

• AD Backend
  • Develop API for receiving and validating feedback data.
  • Pass down the feedback to RCF

---

3. RCF for feedback storage and application

Objective:
Enhance the RCF to incorporate user feedback into its detection process.

Key Tasks:

• RCF:
  • store and reference user feedback.
  • Implement logic to avoid re-reporting false positives or stop missing anomalies that have been labelled
  • Explainability: record an anomaly is suppressed or reported due to label
  • Benchmarking: Test that incorporating feedback leads to improved anomaly detection accuracy.

---

4. Historical Analysis for Threshold Suggestion

Objective:
Collect feedback from historical analysis results to infer and propose threshold adjustments.

Key Tasks:

• AD backend

  • Create a "suggest" API that calculates and proposes adjustments based on historical data. See existing forecasting suggest api for an example. Note the suggestion will change once users modify other parameters like interval and shingle size.
  • Benchmarking: Verify the suggested parameter helps in various data set

• AD frontend

  • Display a callout or notification suggesting rule changes when sufficient feedback is available.
  • Provide visual comparisons of current versus proposed thresholds.

---

5. Real-Time Analysis for Threshold Suggestion

Objective:
Collect feedback from real-time analysis results to adjust thresholds dynamically.

Key Tasks:

• AD backend

  • Develop real-time analysis similar to the historical approach to help adjust thresholds on the fly.

• RCF:

  • Verify that the current RCF implementation supports real-time threshold adjustment. If not, enhance RCF to handle these adjustments.

Timeline

Phase	Duration
Phase 1: Requirements & UX	4 Weeks
Phase 2: Backend Integration	3 Weeks
Phase 3: RCF Enhancements	4 Weeks
Phase 4: Historical Analysis	6 Weeks
Phase 5: Real-Time Analysis	4 Weeks
Phase 6: Testing & Launch & appsec review	6 Weeks