Model Confidence Drift: Low-Confidence Signals and Distribution Windows

What 1-SEC checks

Reported model confidence greater than zero and below 0.1 produces a potential-adversarial-input alert. Predictions accumulate a mean and ten confidence buckets per model. On a subsequent prediction more than one hour after a window starts, the prior nonempty current window becomes the baseline. A model-update event evaluates drift once both baseline and current windows contain at least ten predictions. Absolute mean change above 0.15 or the implemented Jensen-Shannon distribution score above 0.3 marks significant drift.

Read the implementation: handleInference

Required telemetry and context

Relevant input events: model_inference, prediction, model_update, model_deploy, weight_change. Use a trusted producer at the real application boundary.

• model_id
• confidence (number)
• input_hash
• model update event to evaluate accumulated drift

Field names above describe relevant context, not a complete API request. Consult the current event contract for envelope and detail requirements. Preserve booleans, numbers, and strings as the receiving handler expects them. Agent-provided text cannot independently verify identity or authorization.

Deployment, prevention, and response

Report confidence on a consistent numeric scale from the real inference service and separate model IDs and versions according to your deployment. Collect enough trusted baseline observations before evaluating a changed distribution. Investigate traffic changes, class mix and calibration alongside security hypotheses. Validate the whole prediction-to-model-update path in an isolated environment. Keep approved model versions and independent quality checks; the monitor does not roll back a model or safely retrain it.

Record the source event, affected agent/host, actual target, policy decision, and response result. Review enforcement configuration separately from detection. A recommendation, selected action, dry-run record, and completed action are different outcomes.

Reproduce the source fixtures

From a checkout of the linked 1-SEC source revision, run the selected fixtures with the repository's Go toolchain. These commands run tests; they do not execute a real attack against your systems.

go test ./internal/modules/datapoisoning -run '^(TestGuard_HandleEvent_Inference_LowConfidence|TestModelDriftMonitor_CheckDrift_InsufficientData|TestModelDriftMonitor_CheckDrift_Normal|TestModelDriftMonitor_DistributionShift|TestModelDriftMonitor_BucketTracking|TestJSDiv_IdenticalDistributions)$' -count=1

The fixtures test the low-confidence handler, insufficient samples, normal and shifted synthetic statistics, bucket placement and identical-distribution behavior. Drift fixtures seed statistics directly; they are not a wall-clock baseline-rotation integration test or an adversarial-ML benchmark.

func TestGuard_HandleEvent_Inference_LowConfidence(t *testing.T) {
	cp := makeCapturingPipeline()
	g := startedModuleWithPipeline(t, cp)
	defer g.Stop()

	ev := core.NewSecurityEvent("test", "model_inference", core.SeverityInfo, "inference")
	ev.Details["model_id"] = "model-prod"
	ev.Details["confidence"] = 0.05 // very low confidence
	ev.Details["input_hash"] = "abc123"
	ev.SourceIP = "10.0.0.1"

	if err := g.HandleEvent(ev); err != nil {
		t.Fatalf("HandleEvent() error: %v", err)
	}

	if cp.count() == 0 {
		t.Error("expected alert for adversarial input (low confidence)")
	}
	if !cp.hasAlertType("adversarial_input") {
		t.Error("expected adversarial_input alert type")
	}
}

func TestModelDriftMonitor_CheckDrift_InsufficientData(t *testing.T) {
	dm := NewModelDriftMonitor()

	// Only a few predictions — not enough for drift detection
	for i := 0; i < 5; i++ {
		dm.RecordPrediction("model-2", 0.9)
	}

	result := dm.CheckDrift("model-2")
	if result.Significant {
		t.Error("expected no significant drift with insufficient data (<10 samples)")
	}
}

func TestModelDriftMonitor_CheckDrift_Normal(t *testing.T) {
	dm := NewModelDriftMonitor()

	// Small delta — no drift
	dm.models["model-4"] = &modelStats{
		baselineSum:   9.0, // mean = 0.9
		baselineCount: 10,
		currentSum:    8.8, // mean = 0.88
		currentCount:  10,
	}

	result := dm.CheckDrift("model-4")
	if result.Significant {
		t.Error("expected no significant drift for small delta")
	}
}

func TestModelDriftMonitor_DistributionShift(t *testing.T) {
	dm := NewModelDriftMonitor()

	// Baseline: all predictions in high-confidence bucket
	baseline := &modelStats{
		baselineSum:   9.0,
		baselineCount: 10,
		currentSum:    3.0, // mean shifted to 0.3
		currentCount:  10,
	}
	// Baseline distribution: all in bucket 9 (0.9-1.0)
	baseline.baselineBuckets[9] = 10
	// Current distribution: all in bucket 3 (0.3-0.4)
	baseline.currentBuckets[3] = 10

	dm.models["model-dist"] = baseline

	result := dm.CheckDrift("model-dist")
	if !result.Significant {
		t.Error("expected significant drift for distribution shift")
	}
	if result.DistributionShift <= 0 {
		t.Error("expected positive distribution shift value")
	}
}

func TestModelDriftMonitor_BucketTracking(t *testing.T) {
	dm := NewModelDriftMonitor()

	dm.RecordPrediction("model-bucket", 0.95)
	dm.RecordPrediction("model-bucket", 0.35)
	dm.RecordPrediction("model-bucket", 0.05)

	stats := dm.models["model-bucket"]
	if stats.currentBuckets[9] != 1 { // 0.95 → bucket 9
		t.Errorf("expected bucket 9 count=1, got %d", stats.currentBuckets[9])
	}
	if stats.currentBuckets[3] != 1 { // 0.35 → bucket 3
		t.Errorf("expected bucket 3 count=1, got %d", stats.currentBuckets[3])
	}
	if stats.currentBuckets[0] != 1 { // 0.05 → bucket 0
		t.Errorf("expected bucket 0 count=1, got %d", stats.currentBuckets[0])
	}
}

func TestJSDiv_IdenticalDistributions(t *testing.T) {
	a := []int{5, 5, 5, 5, 5, 5, 5, 5, 5, 5}
	result := jsDiv(a, 50, a, 50)
	if result > 0.001 {
		t.Errorf("expected ~0 divergence for identical distributions, got %f", result)
	}
}

Tuning, evaluation, and coverage limits

Low confidence is a heuristic observation, not evidence that an input is adversarial or that a model was poisoned. A zero or missing confidence does not take the low-confidence alert branch. The EventTypes list includes inference_result, but HandleEvent does not dispatch that alias; use model_inference or prediction. Rotation occurs on a later prediction, not on a background scheduler, and a new baseline can include prior suspicious data. Drift evaluation requires a model-update event; recording predictions alone does not emit a drift alert. Thresholds are fixed, state is in memory, and sample fixtures do not calibrate accuracy.

Evaluate this behavior with representative authorized activity and a controlled known-positive case. Investigate the exact rule, inputs, and state before suppressing an alert. Do not assume a hard-coded heuristic is a configurable setting. Measure your own deployment; the source fixtures do not establish a benchmark, detection rate, compliance certification, or incident-prevention claim.