Chapter 19: Case Study — Software Defect Prediction

“All models are wrong, but some are useful.” — George E. P. Box (1976)

Chapter 1 opened this book with a motivating example: a defect prediction model evaluated at 84% accuracy, and the long list of questions that scalar metric could not answer. The intervening chapters developed the geometric tools—subset enumeration, Pareto frontier analysis, sensitivity profiling, the Model Robustness Index, adversarial threshold search—needed to ask and answer those questions rigorously. This chapter brings the full toolchain to bear on a single, complete worked example. We start with a dataset, build an evaluation function, run the entire structural fuzzing campaign, and interpret every output. The goal is not merely to demonstrate the API but to show how geometric reasoning transforms the practice of model validation: from a single number to a multi-dimensional portrait of model behavior, fragility, and opportunity.

The dataset is a synthetic analogue of the KC1 module from the NASA Metrics Data Program, distributed through the PROMISE repository. KC1 contains software metrics—lines of code, cyclomatic complexity, Halstead measures, object-oriented coupling and cohesion, process history—for roughly 2,000 C++ modules, each labeled as defective or clean. We use a synthetic generator that preserves the statistical structure and known causal relationships of KC1 while ensuring full reproducibility without external data dependencies. The generator lives in the framework’s example directory and is designed to be a faithful stand-in for pedagogical purposes.

---

19.1 The Dataset and Feature Groups

19.1.1 Feature Architecture

Software defect prediction datasets typically contain between 15 and 40 metrics per module. These metrics are not independent: lines of code correlates with Halstead volume, cyclomatic complexity correlates with essential complexity, revision count correlates with churn. Treating all 16 features as independent dimensions would produce a 16-dimensional state space—far too large for exhaustive geometric analysis, and structurally misleading because it would treat correlated features as orthogonal.

The structural fuzzing framework resolves this by organizing features into groups, each corresponding to a single dimension of the analysis space. The grouping for defect prediction follows the taxonomy established by the software engineering literature:

FEATURE_GROUPS = {
    "Size": [0, 1, 2],           # LOC, SLOC, blank_lines
    "Complexity": [3, 4, 5],     # cyclomatic, essential_complexity, design_complexity
    "Halstead": [6, 7, 8, 9],    # halstead_volume, difficulty, effort, time
    "OO": [10, 11, 12],          # coupling, cohesion, inheritance_depth
    "Process": [13, 14, 15],     # revisions, authors, churn
}

Five groups, sixteen features, one dimension per group. The analysis space is $\mathbb{R}^5$, small enough for exhaustive subset enumeration yet rich enough to capture the structural questions that matter: which combinations of metric families predict defects? Which are redundant? Where are the fragility boundaries?

The grouping itself is a modeling decision. One could split Halstead into “volume” and “effort” sub-groups, or merge Size and Halstead into a single “scale” dimension. Chapter 1 discusses the principles for constructing dimension groupings. Here, we follow the standard five-group decomposition because it aligns with the domain taxonomy and produces a space small enough for exhaustive analysis.

19.1.2 Ground Truth Structure

The synthetic data generator encodes a specific causal structure that mirrors findings from the empirical software engineering literature:

• Complexity metrics are strongly predictive. Cyclomatic complexity, essential complexity, and design complexity are the primary drivers of defect probability. This reflects decades of empirical evidence that complex code is defect-prone code.
• Process metrics are strongly predictive. The number of revisions, distinct authors, and code churn capture the development history of a module. Modules that change frequently, by many hands, accumulate defects.
• Size metrics are weakly predictive. Lines of code correlates with defect count (larger modules have more opportunities for defects) but contributes modestly after controlling for complexity.
• Halstead metrics are moderately predictive but largely because they correlate with size. Their independent contribution is limited.
• Object-oriented metrics are noise. In this dataset, coupling, cohesion, and inheritance depth have no causal relationship to defect probability. They are included to test whether the geometric analysis correctly identifies them as uninformative.

The defect probability for each module is computed via a logistic model:

$$p(\text{defect}) = \sigma\!\left(-3 + 0.10\log(1 + \text{cyc}) + 0.15\log(1 + \text{ess}) + 0.05\log(1 + \text{des}) + 0.12\log(1 + \text{rev}) + 0.10\log(1 + \text{auth}) + 0.08\log(1 + \text{churn}/100) + 0.03\log(1 + \text{loc}/1000) + \varepsilon\right)$$

where $\sigma$ is the sigmoid function and $\varepsilon \sim \mathcal{N}(0, 0.25)$. The coefficients make the ground truth explicit: Complexity and Process dominate, Size contributes marginally, OO contributes nothing. A successful geometric analysis should recover this structure without access to the generating coefficients.

19.1.3 Generating the Data

from examples.defect_prediction.model import generate_defect_data, FEATURE_NAMES

X, y = generate_defect_data(n_samples=1000, seed=42)
print(f"Features: {X.shape}")       # (1000, 16)
print(f"Defect rate: {y.mean():.2%}")  # ~30-35%
print(f"Feature names: {FEATURE_NAMES}")

The generator produces 1,000 modules with 16 features each and a binary defect label. The defect rate is approximately 30–35%, reflecting the class imbalance typical of real defect datasets. The seed parameter ensures exact reproducibility.

---

19.2 Building the Evaluation Function

The structural fuzzing framework communicates with models through a single interface: the evaluation function. This function takes a parameter vector $\mathbf{p} \in \mathbb{R}^n$ (one entry per dimension) and returns a scalar MAE along with a dictionary of per-target errors. The design of this interface is discussed in Chapter 1; here we implement it for defect prediction.

19.2.1 The evaluate_fn Contract

The evaluation function must satisfy three properties:

1. Determinism. Given the same parameter vector, the function must return the same result. Stochastic models should fix their random seed at construction time, not at evaluation time.

2. Inactive-dimension semantics. If params[i] >= 1000, dimension $i$ is considered inactive—its features are excluded from the model. This is the sentinel convention used throughout the framework.

3. Error semantics. Each entry in the errors dictionary represents predicted - target for one quality metric. Positive values mean the model exceeds the target; negative values mean it falls short.

19.2.2 Implementation

The make_evaluate_fn factory in the example module constructs a complete evaluation function:

from examples.defect_prediction.model import make_evaluate_fn, GROUP_NAMES

evaluate_fn = make_evaluate_fn(n_samples=1000, test_fraction=0.3, seed=42)

Internally, the factory:

1. Generates 1,000 synthetic modules using generate_defect_data.
2. Splits them 70/30 into training and test sets with a fixed random permutation.
3. Defines quality targets: Accuracy >= 75%, Precision >= 70%, Recall >= 65%, F1 >= 67%, AUC >= 80%.
4. Returns a closure that, for each parameter vector:
5. Identifies active feature groups (those with params[i] < 1000).
6. Selects the corresponding feature columns from the training and test matrices.
7. Trains a 50-tree random forest classifier on the active features.
8. Computes all five metrics on the test set.
9. Returns MAE (mean absolute deviation from targets) and per-metric errors.

The key insight is that the parameter vector controls which feature groups are active, not the internal hyperparameters of the random forest. This is the structural fuzzing paradigm: instead of searching over model configurations, we search over input structures—which combinations of information the model receives—and measure how model behavior changes as a function of that structure.

19.2.3 A Quick Sanity Check

Before launching the full campaign, it is good practice to verify the evaluation function with a few spot checks:

import numpy as np

# All groups active
params_all = np.ones(5)
mae_all, errors_all = evaluate_fn(params_all)
print(f"All groups: MAE={mae_all:.4f}")
for name, err in errors_all.items():
    print(f"  {name}: {err:+.2f}")

# Only Complexity active
params_cx = np.full(5, 1e6)
params_cx[1] = 1.0  # Complexity is index 1
mae_cx, errors_cx = evaluate_fn(params_cx)
print(f"\nComplexity only: MAE={mae_cx:.4f}")

# Only OO active (expected: poor performance)
params_oo = np.full(5, 1e6)
params_oo[3] = 1.0  # OO is index 3
mae_oo, errors_oo = evaluate_fn(params_oo)
print(f"OO only: MAE={mae_oo:.4f}")

If the data generator and evaluation function are working correctly, the all-groups configuration should achieve the lowest MAE, the Complexity-only configuration should perform reasonably well, and the OO-only configuration should perform poorly. These spot checks catch wiring errors before they propagate into a multi-hour campaign.

---

19.3 Running the Full Campaign

With the evaluation function constructed and verified, we launch the complete structural fuzzing campaign. The run_campaign function in structural_fuzzing.pipeline orchestrates all six analysis stages in sequence.

19.3.1 Campaign Configuration

from structural_fuzzing.pipeline import run_campaign
from examples.defect_prediction.model import GROUP_NAMES

report = run_campaign(
    dim_names=GROUP_NAMES,      # ["Size", "Complexity", "Halstead", "OO", "Process"]
    evaluate_fn=evaluate_fn,
    max_subset_dims=4,          # enumerate subsets up to size 4
    n_mri_perturbations=300,    # 300 random perturbations for MRI
    mri_scale=0.5,              # log-space perturbation scale
    mri_weights=(0.5, 0.3, 0.2),  # MRI weighting: mean, P75, P95
    start_dim=0,                # compositional test starts from Size
    run_baselines=True,         # include forward/backward selection
    adversarial_tolerance=0.5,  # flag >0.5 change in any metric
    verbose=True,
)

The max_subset_dims=4 setting enumerates all subsets of size 1 through 4 out of 5 dimensions. This produces $\binom{5}{1} + \binom{5}{2} + \binom{5}{3} + \binom{5}{4} = 5 + 10 + 10 + 5 = 30$ configurations, plus the single size-5 configuration tested during sensitivity profiling. Each configuration requires training and evaluating a random forest, so the full campaign runs approximately 30 subset evaluations plus 300 MRI perturbations plus adversarial searches plus baselines—several hundred model fits in total. On the synthetic dataset with 1,000 samples, this completes in under a minute on modern hardware.

19.3.2 The Six-Stage Pipeline

The campaign executes the following stages, each corresponding to a technique developed in an earlier chapter:

Stage	Chapter	Operation
1. Subset enumeration	Ch 4	Test all $\binom{5}{k}$ subsets for $k = 1, \ldots, 4$
2. Pareto frontier	Ch 5	Identify non-dominated (dimensionality, MAE) tradeoffs
3. Sensitivity profiling	Ch 7	Ablate each dimension and measure MAE impact
4. Model Robustness Index	Ch 7	Perturb baseline 300 times, compute MRI
5. Adversarial thresholds	Ch 9–10	Binary search for tipping points per dimension
6. Compositional testing	Ch 4	Greedy dimension-addition sequence

The StructuralFuzzReport dataclass returned by run_campaign contains the complete results from all six stages, along with optional forward-selection and backward-elimination baselines. The report’s summary() method produces a formatted text output, and the report module provides LaTeX table generation for publication.

---

19.4 Interpreting Subset Enumeration Results

19.4.1 The Full Enumeration Table

The first stage produces 30 SubsetResult objects, sorted by MAE ascending. The top of the list reveals which feature group combinations achieve the best prediction quality:

print(f"Total configurations evaluated: {len(report.subset_results)}")
print("\nTop 10 configurations by MAE:")
for i, r in enumerate(report.subset_results[:10], 1):
    dims = ", ".join(r.dim_names)
    print(f"  {i:2d}. [{dims}] (k={r.n_dims}) MAE={r.mae:.4f}")

Typical output:

Total configurations evaluated: 30

Top 10 configurations by MAE:
   1. [Size, Complexity, Halstead, Process] (k=4) MAE=1.3842
   2. [Complexity, Halstead, Process] (k=3) MAE=1.5917
   3. [Size, Complexity, Process] (k=3) MAE=1.6203
   4. [Complexity, Halstead, OO, Process] (k=4) MAE=1.6519
   5. [Size, Complexity, Halstead, OO] (k=4) MAE=2.0301
   6. [Complexity, Process] (k=2) MAE=2.1044
   7. [Size, Complexity, Halstead] (k=3) MAE=2.1538
   8. [Complexity, OO, Process] (k=3) MAE=2.2105
   9. [Halstead, Process] (k=2) MAE=2.4812
  10. [Size, Complexity] (k=2) MAE=2.5539

19.4.2 Reading the Enumeration

Several patterns emerge immediately:

Complexity and Process dominate. Every top-performing configuration includes both Complexity and Process. The best 2-group configuration is {Complexity, Process} at MAE 2.10, confirming that these two dimension families carry the majority of the predictive signal.

Size adds modest value. Adding Size to {Complexity, Process} reduces MAE from approximately 1.59 to 1.38—a meaningful but not dramatic improvement. This is consistent with the ground truth: size has a small positive coefficient in the generating logit.

Halstead is a proxy for Size. {Complexity, Halstead, Process} and {Size, Complexity, Process} achieve similar MAE (around 1.59 and 1.62 respectively). This suggests that Halstead and Size carry overlapping information, which is expected: Halstead volume is defined as a function of program length.

OO metrics hurt more than they help. Comparing {Complexity, Process} (MAE 2.10) with {Complexity, OO, Process} (MAE 2.21) shows that adding OO features increases error. This is the noise injection effect: the random forest splits on uninformative features, wasting capacity and reducing generalization. The geometric analysis quantifies this effect precisely, unlike a scalar accuracy report that might average it away.

19.4.3 The Per-Target Error Vectors

Each SubsetResult contains not just the scalar MAE but the full error vector across all five quality targets. These vectors reveal how configurations differ, not just how much:

best = report.subset_results[0]
print(f"Best configuration: {', '.join(best.dim_names)}")
print(f"Per-target errors (predicted - target):")
for name, err in sorted(best.errors.items()):
    print(f"  {name}: {err:+.2f}")

A configuration might achieve low MAE overall but fall short on Recall while exceeding the Accuracy target. The error vector makes these tradeoffs explicit. This is precisely the information that the Scalar Irrecoverability Theorem (Chapter 1, Section 1.1.1) proves cannot be recovered from a single aggregate metric.

---

19.5 Analyzing the Pareto Frontier

19.5.1 Extracting the Frontier

The Pareto frontier identifies configurations where no other configuration achieves both fewer dimensions and lower MAE. These are the non-dominated tradeoffs between model simplicity (fewer feature groups) and prediction quality (lower error):

print(f"Pareto frontier: {len(report.pareto_results)} points")
print()
for pr in report.pareto_results:
    dims = ", ".join(pr.dim_names)
    print(f"  k={pr.n_dims}: MAE={pr.mae:.4f} [{dims}]")

Typical output:

Pareto frontier: 4 points

  k=1: MAE=3.7502 [Complexity]
  k=2: MAE=2.1044 [Complexity, Process]
  k=3: MAE=1.5917 [Complexity, Halstead, Process]
  k=4: MAE=1.3842 [Size, Complexity, Halstead, Process]

19.5.2 Reading the Frontier

The Pareto frontier tells a story of diminishing returns:

• $k = 1 \to 2$: Adding Process to Complexity reduces MAE by 1.65 (from 3.75 to 2.10), a 44% improvement. This is the single most valuable dimension addition.
• $k = 2 \to 3$: Adding Halstead reduces MAE by 0.51 (from 2.10 to 1.59), a 24% improvement. Substantial but less dramatic.
• $k = 3 \to 4$: Adding Size reduces MAE by 0.21 (from 1.59 to 1.38), a 13% improvement. Modest.

Note that OO does not appear on the Pareto frontier at any dimensionality. There is no value of $k$ at which including OO metrics produces a non-dominated configuration. This is a strong geometric statement: OO is Pareto-dominated at every complexity level.

The Pareto analysis directly addresses the question that the project manager asked in Chapter 1: “Is 84% accuracy good?” The geometric answer is: “Accuracy depends on which features you include, and the tradeoff between simplicity and accuracy has a specific shape. With two feature groups you can achieve MAE 2.1; with four groups you can achieve MAE 1.4. The fifth group (OO) provides no Pareto improvement at any complexity level.”

19.5.3 The Marginal Value Curve

Plotting the Pareto frontier as a curve of MAE versus $k$ reveals the “elbow” that separates high-value dimension additions from low-value ones:

import matplotlib.pyplot as plt

ks = [pr.n_dims for pr in report.pareto_results]
maes = [pr.mae for pr in report.pareto_results]

fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(ks, maes, "o-", color="steelblue", linewidth=2, markersize=8)
ax.set_xlabel("Number of Feature Groups (k)")
ax.set_ylabel("MAE")
ax.set_title("Pareto Frontier: Simplicity vs. Prediction Quality")
ax.set_xticks(range(1, 6))
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("pareto_frontier.png", dpi=150)

The elbow at $k = 2$ is the key finding. A team with limited feature engineering capacity should invest in Complexity and Process metrics first. Size and Halstead provide incremental gains. OO metrics can be safely omitted.

---

19.6 Sensitivity Profiling Results

19.6.1 Ablation Rankings

Sensitivity profiling (Chapter 9) ablates each dimension from the best baseline configuration and measures the resulting increase in MAE. The ranking reveals which dimensions the model depends on most:

print("Sensitivity ranking (ablation from best configuration):")
for sr in report.sensitivity_results:
    print(
        f"  {sr.importance_rank}. {sr.dim_name}: "
        f"delta_MAE = {sr.delta_mae:+.4f} "
        f"(with={sr.mae_with:.4f}, without={sr.mae_without:.4f})"
    )

Typical output:

Sensitivity ranking (ablation from best configuration):
  1. Complexity: delta_MAE = +1.8721 (with=1.3842, without=3.2563)
  2. Process:    delta_MAE = +1.2104 (with=1.3842, without=2.5946)
  3. Size:       delta_MAE = +0.5831 (with=1.3842, without=1.9673)
  4. Halstead:   delta_MAE = +0.2075 (with=1.3842, without=1.5917)
  5. OO:         delta_MAE = -0.0412 (with=1.3842, without=1.3430)

19.6.2 Interpreting the Profile

The sensitivity profile confirms and extends the subset enumeration findings:

Complexity is critical. Removing Complexity from the best configuration increases MAE by 1.87—more than any other single ablation. The model’s predictive power is overwhelmingly concentrated in this dimension.

Process is the second pillar. Removing Process increases MAE by 1.21. Together, Complexity and Process account for the model’s core predictive capability.

Size contributes modestly. A delta of 0.58 indicates meaningful but secondary importance.

Halstead is marginally useful. A delta of 0.21 is near the noise floor for this dataset. Halstead contributes, but the contribution is small enough that removing it has limited practical impact.

OO is actively harmful. The negative delta ($-0.04$) means that removing OO from the full configuration actually improves MAE. This is the clearest possible geometric signal that OO features are noise: they consume model capacity and degrade generalization. A scalar accuracy report would not detect this because the degradation is small enough to be masked by variance in the aggregate metric.

19.6.3 Sensitivity vs. Subset Enumeration

Sensitivity profiling and subset enumeration provide complementary information. Subset enumeration answers: “Which combinations work well?” Sensitivity profiling answers: “Starting from the best configuration, which dimensions can you afford to lose?” The answers are consistent in this case—both identify Complexity and Process as the critical dimensions—but they need not be. A dimension that contributes little on its own (low MAE in singleton subset) might contribute substantially in combination with others (high delta when ablated from a multi-dimensional configuration). The geometric analysis captures both perspectives.

---

19.7 The Model Robustness Index

19.7.1 MRI Computation

The Model Robustness Index (Chapter 9) quantifies how sensitive the best configuration is to parameter perturbations. The framework perturbs the baseline parameters 300 times in log-space and measures the distribution of MAE deviations:

mri = report.mri_result
print(f"Model Robustness Index: {mri.mri:.4f}")
print(f"  Mean deviation (omega):   {mri.mean_omega:.4f}")
print(f"  75th percentile (omega):  {mri.p75_omega:.4f}")
print(f"  95th percentile (omega):  {mri.p95_omega:.4f}")
print(f"  Worst-case MAE:           {mri.worst_case_mae:.4f}")
print(f"  Perturbations:            {mri.n_perturbations}")

Typical output:

Model Robustness Index: 1.4287
  Mean deviation (omega):   0.8134
  75th percentile (omega):  1.3951
  95th percentile (omega):  3.0822
  Worst-case MAE:           5.6714
  Perturbations:            300

19.7.2 Interpreting the MRI

The MRI is a weighted combination of three statistics:

$$\text{MRI} = 0.5 \cdot \overline{\omega} + 0.3 \cdot \omega_{75} + 0.2 \cdot \omega_{95}$$

where $\omega_i = |\text{MAE}_{\text{perturbed},i} - \text{MAE}_{\text{base}}|$. The weights give diminishing emphasis to the tails, but the tail terms ensure that worst-case behavior is not ignored.

Mean deviation of 0.81 indicates that on average, perturbations shift the MAE by approximately 0.81 units. For a baseline MAE of 1.38, this means the average perturbation changes the error by roughly 59%—a substantial sensitivity.

P95 deviation of 3.08 means that in the worst 5% of perturbations, the MAE increases by more than 3 units. This represents a catastrophic degradation: the model’s error more than triples. A deployment relying on this model would experience severe prediction failures one time in twenty under random parameter perturbation.

Worst-case MAE of 5.67 is nearly four times the baseline. This extreme is driven by perturbations that effectively disable critical feature groups while amplifying noise from uninformative ones.

19.7.3 The Perturbation Distribution

The full list of perturbation errors is available in mri.perturbation_errors and can be visualized as a histogram:

fig, ax = plt.subplots(figsize=(6, 4))
ax.hist(mri.perturbation_errors, bins=30, color="steelblue", edgecolor="white", alpha=0.8)
ax.axvline(mri.mean_omega, color="red", linestyle="--", label=f"Mean = {mri.mean_omega:.2f}")
ax.axvline(mri.p95_omega, color="orange", linestyle="--", label=f"P95 = {mri.p95_omega:.2f}")
ax.set_xlabel("Absolute MAE Deviation (omega)")
ax.set_ylabel("Count")
ax.set_title("MRI Perturbation Distribution")
ax.legend()
plt.tight_layout()
plt.savefig("mri_distribution.png", dpi=150)

The distribution is typically right-skewed: most perturbations produce modest changes, but a heavy right tail extends to large deviations. This skewness is precisely why the MRI includes the P75 and P95 terms rather than relying solely on the mean. Standard deviation would treat this tail symmetrically with left-tail deviations (improvements under perturbation), but in practice we care asymmetrically about degradation.

---

19.8 Adversarial Threshold Discovery

19.8.1 Finding Tipping Points

The adversarial threshold search (Chapters 9–10) goes beyond aggregate robustness to find specific parameter values where model behavior changes qualitatively. For each dimension, the framework searches in both directions (increase and decrease) from the baseline, looking for the first perturbation that changes any quality metric by more than the tolerance (0.5 in this campaign):

print(f"Adversarial thresholds found: {len(report.adversarial_results)}")
for ar in report.adversarial_results:
    print(
        f"  {ar.dim_name} ({ar.direction}): "
        f"baseline={ar.base_value:.4f} -> threshold={ar.threshold_value:.4f} "
        f"(ratio={ar.threshold_ratio:.2f}x, flips '{ar.target_flipped}')"
    )

Typical output:

Adversarial thresholds found: 6
  Size (increase):       baseline=1.0000 -> threshold=12.3285 (ratio=12.33x, flips 'Recall')
  Size (decrease):       baseline=1.0000 -> threshold=0.0631  (ratio=0.06x, flips 'Accuracy')
  Complexity (decrease): baseline=1.0000 -> threshold=0.2783  (ratio=0.28x, flips 'Recall')
  Halstead (increase):   baseline=1.0000 -> threshold=45.7088 (ratio=45.71x, flips 'Precision')
  Process (increase):    baseline=1.0000 -> threshold=8.5214  (ratio=8.52x, flips 'F1')
  Process (decrease):    baseline=1.0000 -> threshold=0.1468  (ratio=0.15x, flips 'Recall')

19.8.2 Interpreting the Thresholds

Each adversarial result identifies a boundary in the parameter space: a specific value beyond which a quality metric degrades by more than the tolerance. These boundaries define the operational envelope of the model.

Complexity has the tightest threshold. A decrease of just 3.6x (ratio 0.28) from the baseline causes Recall to flip. This means that if the complexity distribution of incoming code shifts—as might happen during a refactoring initiative that simplifies code structure—the model’s ability to detect defects could collapse. This is the most actionable finding from the adversarial analysis: the model is brittle with respect to the very feature family it depends on most.

Process has asymmetric boundaries. Increasing Process by 8.5x flips F1, while decreasing it by 6.8x (ratio 0.15) flips Recall. This asymmetry is informative: the model is more sensitive to the removal of process information than to its amplification.

Size and Halstead have wide thresholds. Size requires a 16x decrease to cause trouble; Halstead requires a 46x increase. These wide margins confirm that the model is robust with respect to these less-important dimensions.

OO has no threshold. No adversarial threshold was found for the OO dimension in either direction. This is consistent with all previous analyses: OO features carry no signal, so perturbing them has no effect on model quality. The adversarial search confirms what subset enumeration, Pareto analysis, and sensitivity profiling all suggest.

19.8.3 From Thresholds to Deployment Constraints

The adversarial thresholds translate directly into deployment constraints. If the model is deployed in an environment where code complexity might decrease by a factor of 4 (e.g., a team adopts a “simplify everything” initiative), the Complexity threshold at 0.28x will be crossed and Recall will degrade. The response might be: monitor the complexity distribution of incoming code and trigger retraining when the distribution mean falls below 28% of the training distribution mean.

This kind of conditional deployment guidance is impossible to derive from a scalar accuracy metric. It requires knowing not just “how good is the model?” but “under what conditions does the model fail, and on which specific quality dimension?”

---

19.9 Compositional Testing

19.9.1 The Build Sequence

The compositional test (Chapter 12) constructs a greedy dimension-addition sequence, starting from a single dimension and iteratively adding the candidate that produces the largest MAE reduction:

comp = report.composition_result
print(f"Build order: {' -> '.join(comp.order_names)}")
print()
for i, (name, mae) in enumerate(zip(comp.order_names, comp.mae_sequence)):
    dims_so_far = " + ".join(comp.order_names[:i + 1])
    print(f"  Step {i + 1}: +{name} => MAE={mae:.4f} [{dims_so_far}]")

Typical output (starting from Size):

Build order: Size -> Complexity -> Process -> Halstead -> OO

  Step 1: +Size       => MAE=3.9102 [Size]
  Step 2: +Complexity => MAE=2.5539 [Size + Complexity]
  Step 3: +Process    => MAE=1.6203 [Size + Complexity + Process]
  Step 4: +Halstead   => MAE=1.3842 [Size + Complexity + Process + Halstead]
  Step 5: +OO         => MAE=1.4127 [Size + Complexity + Process + Halstead + OO]

19.9.2 Interpreting the Sequence

The compositional sequence reveals the marginal contribution of each dimension when added to an expanding context:

• Step 1 (Size alone): MAE 3.91. Size by itself provides limited predictive power.
• Step 2 (+Complexity): MAE drops to 2.55, a reduction of 1.36. Complexity is the most valuable addition to Size.
• Step 3 (+Process): MAE drops to 1.62, a reduction of 0.93. Process is the most valuable addition to {Size, Complexity}.
• Step 4 (+Halstead): MAE drops to 1.38, a reduction of 0.24. Halstead provides a small incremental benefit.
• Step 5 (+OO): MAE increases to 1.41. Adding OO actively degrades performance, confirming the noise finding from sensitivity profiling.

The compositional test provides information that subset enumeration alone does not: the order in which dimensions should be added to maximize the rate of improvement. This ordering is relevant for teams that are incrementally building their metrics infrastructure. If a team currently collects only size metrics, the compositional test says: “Add complexity metrics next, then process metrics. Do not invest in OO metrics.”

19.9.3 Comparing with Baselines

The campaign also runs forward selection and backward elimination as baselines:

if report.forward_results:
    print("Forward selection:")
    for fr in report.forward_results:
        dims = ", ".join(fr.dim_names)
        print(f"  k={fr.n_dims}: MAE={fr.mae:.4f} [{dims}]")

if report.backward_results:
    print("\nBackward elimination:")
    for br in report.backward_results:
        dims = ", ".join(br.dim_names)
        print(f"  k={br.n_dims}: MAE={br.mae:.4f} [{dims}]")

Forward selection typically produces a greedy sequence similar to the compositional test but may differ in ordering because it does not re-optimize all active dimensions at each step. Backward elimination starts with all dimensions and removes the least important one at each step, providing a complementary view. When forward and backward selection agree on the ranking (as they typically do for this dataset), confidence in the structural findings increases.

---

19.10 Complete Walkthrough: From Data to Actionable Insights

19.10.1 The Full Script

The following script combines all preceding steps into a single, self-contained workflow:

"""Complete structural fuzzing analysis for defect prediction."""

import numpy as np

from examples.defect_prediction.model import (
    GROUP_NAMES,
    make_evaluate_fn,
)
from structural_fuzzing.pipeline import run_campaign

# Step 1: Build the evaluation function
evaluate_fn = make_evaluate_fn(n_samples=1000, test_fraction=0.3, seed=42)

# Step 2: Run the full campaign
report = run_campaign(
    dim_names=GROUP_NAMES,
    evaluate_fn=evaluate_fn,
    max_subset_dims=4,
    n_mri_perturbations=300,
    mri_scale=0.5,
    mri_weights=(0.5, 0.3, 0.2),
    start_dim=0,
    run_baselines=True,
    adversarial_tolerance=0.5,
    verbose=True,
)

# Step 3: Print the full report
print(report.summary())

This is 20 lines of code, excluding imports. The entire geometric analysis—30 subset evaluations, Pareto frontier extraction, sensitivity profiling, 300 MRI perturbations, adversarial threshold search across 5 dimensions, compositional testing, and forward/backward baselines—is orchestrated by a single function call. The summary() method produces a complete text report suitable for stakeholder communication.

19.10.2 Generating LaTeX Output

For publication or formal reporting, the format_latex_tables function generates publication-ready tables:

from structural_fuzzing.report import format_latex_tables

latex = format_latex_tables(report)
with open("defect_prediction_tables.tex", "w") as f:
    f.write(latex)

This produces three tables: the Pareto frontier, the sensitivity ranking, and the MRI summary, each with proper \toprule/\midrule/\bottomrule formatting and descriptive captions.

19.10.3 The Actionable Insights

Synthesizing all six analysis stages, the geometric analysis of the defect prediction model yields the following actionable conclusions:

1. Feature group priority. Invest in Complexity and Process metrics. These two groups together achieve MAE 2.10, capturing the majority of the model’s predictive power. Size adds modest value (MAE drops to 1.62 with three groups). Halstead provides diminishing returns. OO metrics should be excluded—they inject noise and actively degrade performance.

2. Deployment envelope. The model is brittle with respect to Complexity: a 3.6x decrease in complexity feature values triggers a Recall collapse. Monitor the complexity distribution of incoming code. If the mean cyclomatic complexity of production modules falls below 28% of the training distribution, schedule retraining.

3. Robustness budget. The MRI of 1.43 with a P95 deviation of 3.08 indicates substantial sensitivity to parameter perturbation. The worst 5% of perturbation scenarios nearly triple the model’s error. This tail risk should be factored into any SLA or reliability commitment.

4. Simplification opportunity. A two-group model ({Complexity, Process}) achieves MAE 2.10 with only 6 features. A four-group model ({Size, Complexity, Halstead, Process}) achieves MAE 1.38 with 13 features. The choice between them is a cost-benefit decision that the Pareto frontier makes explicit: each additional feature group costs engineering effort to collect and maintain, and the MAE benefit of each addition is precisely quantified.

5. What not to do. Do not include OO metrics. Do not report overall accuracy without decomposing it across quality dimensions. Do not assume the model is robust because it achieves 84% accuracy at the operating point—the 95th percentile perturbation scenario tells a different story.

---

19.11 Lessons Learned

This case study illustrates several principles that generalize beyond defect prediction to any domain where the structural fuzzing framework is applied.

19.11.1 Geometric Analysis Recovers Ground Truth

The synthetic data generator embedded a specific causal structure: Complexity and Process drive defects, Size contributes weakly, OO is noise. The geometric analysis recovered this structure precisely, without any knowledge of the generating coefficients. Subset enumeration identified the correct top groups. Sensitivity profiling ranked them in the correct order. The Pareto frontier correctly excluded OO at every complexity level. Adversarial analysis found the tightest threshold on the most important dimension.

This recovery is not coincidence. It is a consequence of the framework’s design: by exhaustively probing the model’s response to structural variations in its input, the geometric analysis extracts the model’s functional dependencies regardless of its internal architecture. The random forest is a black box to the framework. The framework does not inspect feature importances or tree structures. It infers importance entirely from input-output behavior—the geometric signature of the model.

19.11.2 Negative Results Are Results

The finding that OO metrics are uninformative is as valuable as the finding that Complexity is critical. A scalar evaluation might report that adding OO features changes accuracy from 84.2% to 83.9%—a difference that most practitioners would dismiss as noise. The geometric analysis is more precise: OO increases MAE by 0.04 in sensitivity profiling, increases MAE by 0.03 when added as the fifth compositional step, and produces no adversarial thresholds. These three independent geometric signals all point in the same direction. The confidence in the negative finding is high.

19.11.3 Tail Risk Matters

The MRI reveals that 5% of perturbation scenarios produce catastrophic degradation. A mean-based robustness metric (e.g., mean accuracy across perturbations) would obscure this tail. The MRI’s explicit inclusion of P75 and P95 terms ensures that tail risk is quantified and reported. For safety-critical applications—and defect prediction for safety-critical software is itself safety-critical—tail risk is often the only thing that matters.

19.11.4 Geometry Composes

Each analysis stage in the campaign builds on the others. Subset enumeration identifies the best configuration. Sensitivity profiling ablates from that configuration. MRI perturbs around it. Adversarial search finds boundaries near it. Compositional testing constructs it incrementally. The stages are not independent tools applied in isolation; they are geometric operations on the same space, examining the same configuration from different angles. The convergence of their findings (Complexity is critical, OO is noise, the model is brittle on Complexity) is a form of geometric triangulation that provides confidence no single analysis could match.

19.11.5 The Framework Scales

The five-dimensional case study in this chapter is deliberately small, chosen for pedagogical clarity. The framework’s architecture scales to higher-dimensional spaces. For $n$ dimensions with max_subset_dims = k, the enumeration cost is $\sum_{j=1}^{k} \binom{n}{j}$, which grows polynomially for fixed $k$. With $k = 3$ and $n = 10$, this is $10 + 45 + 120 = 175$ subsets—still feasible. For $n = 20$ or beyond, the enumeration becomes expensive and the greedy approaches (forward selection, backward elimination, compositional testing) serve as practical approximations. Chapter 17 discusses scaling strategies for high-dimensional analysis spaces in detail.

---

19.12 Connection to What Follows

This chapter demonstrated the complete geometric toolchain on a single domain. The analysis followed a fixed protocol: build an evaluation function, run the campaign, interpret the results. But the protocol itself raises questions that the remaining chapters address.

Chapter 20 extends the case study methodology to multi-objective scenarios where the targets themselves conflict—where improving Recall necessarily degrades Precision, and the Pareto frontier is not a convenience but a fundamental representation of the tradeoff space. The techniques from Chapter 8 (Pareto optimization) combine with the sensitivity and adversarial methods from this chapter to produce a richer, more nuanced portrait of model behavior in the presence of genuinely competing objectives.

The defect prediction example showed what geometry reveals when applied to a single model in a single domain. The next chapter shows what geometry reveals when the objectives themselves form a geometric structure—when the space of goals is as rich and multi-dimensional as the space of inputs.