Skip to content

feat: add Analysis Mode setting for Security Agent#435

Merged
jeanduplessis merged 6 commits intomainfrom
jdp/security-agent-analysis-mode
Feb 24, 2026
Merged

feat: add Analysis Mode setting for Security Agent#435
jeanduplessis merged 6 commits intomainfrom
jdp/security-agent-analysis-mode

Conversation

@kilo-code-bot
Copy link
Contributor

@kilo-code-bot kilo-code-bot bot commented Feb 22, 2026

Summary

20260224-105311@2x

Adds an analysis_mode setting to the Security Agent configuration that lets users control the depth of vulnerability analysis. Three modes are available:

  • Auto (default): Current behavior — triage runs first, sandbox analysis runs only if triage determines it's needed.
  • Shallow: Triage only — never proceeds to sandbox analysis, regardless of triage output.
  • Deep: Always forces sandbox analysis (leverages the existing forceSandbox mechanism).

Key Design Decisions

  • No database migration needed: The analysis_mode field fits inside the existing JSONB config column on agent_configs. The default-merge pattern in getSecurityAgentConfig() means all existing users automatically get 'auto' mode.
  • Leverages existing forceSandbox: The deep mode reuses the existing sandbox-forcing mechanism.
  • Shallow mode overrides forceSandbox: Even if forceSandbox is explicitly set, shallow mode will never run sandbox analysis.

Changes

Config Schema & Defaults

  • Added AnalysisMode type ('auto' | 'shallow' | 'deep') to src/lib/security-agent/core/types.ts
  • Added analysis_mode field to SecurityAgentConfigSchema with default 'auto'
  • Added AnalysisModeSchema to src/lib/security-agent/core/schemas.ts
  • Updated DEFAULT_SECURITY_AGENT_CONFIG in src/lib/security-agent/core/constants.ts

Analysis Pipeline

  • Updated startSecurityAnalysis() in src/lib/security-agent/services/analysis-service.ts to accept and respect analysisMode
  • Pipeline logic: deep → always sandbox; shallow → never sandbox; auto → respect triage recommendation

API / Routers

  • Both security-agent-router.ts (personal) and organization-security-agent-router.ts (org) now:
    • Return analysisMode from getConfig
    • Accept analysisMode in saveConfig
    • Read analysis_mode from config when starting analysis

UI

  • Added radio group control in SecurityConfigForm.tsx with three options and descriptive help text
  • Wired through SecurityAgentPageClient.tsx

Tracking

  • Added analysisMode to PostHog config-saved event

Tests

  • Added comprehensive tests for all three modes in analysis-service.test.ts:
    • Auto mode: runs sandbox when triage recommends it
    • Auto mode: skips sandbox when triage says not needed
    • Shallow mode: never runs sandbox even when triage recommends it
    • Shallow mode: never runs sandbox even with forceSandbox
    • Deep mode: always runs sandbox even when triage says not needed
    • Defaults to auto mode when analysisMode is not specified

Rollout

Ships behind existing Security Agent feature flag. Default 'auto' means no behavioral change for existing users.


Built for jean by Kilo for Slack

@kilo-code-bot
Copy link
Contributor Author

kilo-code-bot bot commented Feb 22, 2026

Code Review Summary

Status: No Issues Found | Recommendation: Merge

This PR cleanly replaces the forceSandbox boolean with a three-value analysisMode enum (auto / shallow / deep). The refactor is consistent across all layers:

  • Core types/schemasAnalysisMode type and Zod schemas added with .default('auto') for backward compatibility
  • Analysis service — Decision logic correctly maps modes: deep → always sandbox, shallow → never sandbox, auto → triage-driven
  • Routers — Both personal and org routers fetch config unconditionally (needed for analysis_mode) and pass it through correctly
  • UISecurityConfigForm adds a RadioGroup with clear descriptions; follows existing patterns (as const, as AnalysisMode)
  • PostHog trackingforceSandbox: booleananalysisMode?: string in event types
  • Tests — Good coverage of all three modes plus default behavior
  • No migration needed — Config is stored as JSON blob; Zod default handles existing configs
Other Observations (not in diff)

Issues found in unchanged code that cannot receive inline comments:

File Line Issue
src/lib/security-agent/services/analysis-service.ts 6 SUGGESTION: Module-level doc comment still says "only if needed or forced" — should be updated to reference analysisMode
Files Reviewed (10 files)
  • src/components/security-agent/SecurityAgentPageClient.tsx - 0 issues
  • src/components/security-agent/SecurityConfigForm.tsx - 0 issues
  • src/lib/security-agent/core/constants.ts - 0 issues
  • src/lib/security-agent/core/schemas.ts - 0 issues
  • src/lib/security-agent/core/types.ts - 0 issues
  • src/lib/security-agent/posthog-tracking.ts - 0 issues
  • src/lib/security-agent/services/analysis-service.test.ts - 0 issues
  • src/lib/security-agent/services/analysis-service.ts - 0 issues
  • src/routers/organizations/organization-security-agent-router.ts - 0 issues
  • src/routers/security-agent-router.ts - 0 issues

@jeanduplessis jeanduplessis changed the base branch from main to jdp/security-agent-next-phase-2 February 22, 2026 15:22
@jeanduplessis jeanduplessis force-pushed the jdp/security-agent-analysis-mode branch from 83f24f4 to 6a801e5 Compare February 22, 2026 15:24
@jeanduplessis jeanduplessis force-pushed the jdp/security-agent-next-phase-2 branch 2 times, most recently from 1b9483f to 287f248 Compare February 23, 2026 22:45
Base automatically changed from jdp/security-agent-next-phase-2 to main February 23, 2026 23:12
kilo-code-bot bot and others added 6 commits February 24, 2026 10:39
Add a configurable analysis_mode setting with three options:
- auto (default): triage runs first, sandbox only if triage recommends it
- shallow: triage only, never runs sandbox analysis
- deep: always forces sandbox analysis

Changes:
- Add AnalysisMode type and analysis_mode to SecurityAgentConfig schema
- Update DEFAULT_SECURITY_AGENT_CONFIG with analysis_mode: 'auto'
- Add AnalysisModeSchema and analysisMode to SaveSecurityConfigInputSchema
- Update startSecurityAnalysis to respect analysis_mode parameter
- Update both personal and org routers (getConfig, saveConfig, startAnalysis)
- Add Analysis Mode card to SecurityConfigForm UI with radio group
- Pass analysisMode through SecurityAgentPageClient
- Add analysisMode to PostHog tracking events
- Add comprehensive tests for all three modes

No database migration needed - analysis_mode fits in existing JSONB config
column. Default-merge pattern ensures backward compatibility.
forceSandbox was never set to true by any caller and is fully superseded
by analysisMode: 'deep'. Removing it simplifies the sandbox decision to
a clean two-term expression and eliminates the confusing precedence issue
where shallow mode silently ignored an explicit forceSandbox: true.
…ling

Move startSecurityAnalysis result from untyped 'let' outside try/catch to
'const' inside the try block. This eliminates the possibility of accessing
an undefined result if rethrowAsPaymentRequired's 'never' return type is
ever lost in a refactor.
Add analysisMode to SecurityAgentAnalysisStartedEvent so analytics can
distinguish input mode (auto/shallow/deep) from outcome (triageOnly).
Without this, 'auto chose triage-only' and 'shallow forced triage-only'
are indistinguishable in metrics.
@jeanduplessis jeanduplessis force-pushed the jdp/security-agent-analysis-mode branch from a9df5bd to 966713a Compare February 24, 2026 08:46
@jeanduplessis jeanduplessis merged commit 440d896 into main Feb 24, 2026
12 checks passed
@jeanduplessis jeanduplessis deleted the jdp/security-agent-analysis-mode branch February 24, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants