# Experiments — Improvement & Growth Plan

> Package: capell-app/experiments · Kind: package · Tier: premium · Product group: Capell Growth · Bundle: growth · Status: Complete

## 1. Snapshot

Experiments is a server-side A/B testing engine for Capell built entirely from Laravel Actions over six tables (`experiments`, `experiment_variants`, `experiment_goals`, `experiment_audience_rules`, `experiment_allocations`, `experiment_goal_events`). Domain logic lives in `src/Actions`: `CreateExperimentAction` (aggregate writer), `AllocateVariantAction` (deterministic SHA-256 weighted bucketing + sticky allocation row), `ResolveExperimentVariantForContextAction` (candidate query → allocate → `ResolvedExperimentVariantData` with cache-vary metadata), `EvaluateAudienceRulesAction` (path/query/utm/referrer/attribute/segment operators), `RecordGoalEventAction`, `BuildWinnerReportAction`, and `DeclareExperimentWinnerAction`. Surface delivery is admin-only: four Filament resources (Experiment/Variant/Goal/AudienceRule) plus the result page contributed via `ExperimentsServiceProvider::registerAdminResources()`; depends on `capell-app/admin`, `capell-app/core`, Filament, `spatie/laravel-data`, `lorisleiva/laravel-actions`. Current marketplace summary verbatim: _"Run A/B tests with audience rules, sticky variant allocation, goal tracking, and winner reports."_ The manifest now promotes four runner-backed Capell admin screenshots for list/edit/rules/results surfaces.

## Completed Improvement Slices

- **2026-06-03:** Implemented real `ExperimentsHealthCheck` diagnostics and filtered request-context candidate resolution by `subject_class`.
- **2026-06-04:** Honoured `allocation_strategy` in `AllocateVariantAction`, added allocation ids to allocation data, gated winner reports/declarations behind a minimum sample size plus two-proportion significance check, and reconciled manifest/Composer copy with the shipped admin-only surface and statistical-significance capability.
- **2026-06-05:** Made `ResolveExperimentVariantForContextAction` record a non-cacheable frontend render contribution with visitor/experiment/variant vary metadata, so HTML Cache will not persist one visitor's resolved variant as shared anonymous HTML.
- **2026-06-08:** Connected keyed experiment goal events into Insights conversion reporting when the allocation carries an Insights visit uuid, preserving idempotency and source-package metadata for Growth bundle dashboards.

## 2. Improvements (existing functionality)

- **Shipped 2026-06-04: Honour `allocation_strategy` or delete it.** `AllocateVariantAction` now branches on `AllocationStrategy`: `StickyWeighted` reuses a persisted allocation row for the visitor hash, while `Weighted` records a fresh weighted allocation without using sticky lookup state. Allocation data now includes the persisted allocation id so downstream goal/event callers can keep an explicit Action/Data boundary. `src/Actions/AllocateVariantAction.php`, `src/Data/VariantAllocationData.php`, `tests/Feature/ExperimentFoundationTest.php`. (M)
- **Shipped 2026-06-03: Filter `candidateQuery` by `subject_class`.** `ResolveExperimentVariantForContextAction::candidateQuery()` now filters by `subject_class` while preserving class-agnostic experiments, preventing collisions between experiments that share `subject_type` + `subject_id` across different subject classes. `src/Actions/ResolveExperimentVariantForContextAction.php`, `tests/Feature/ResolveVariantSubjectClassTest.php`. (S)
- **Shipped 2026-06-07: Goal-event idempotency guard.** `RecordGoalEventAction` now uses `firstOrCreate` when callers provide an `event_key`, backed by a unique `(experiment_allocation_id, experiment_goal_id, event_key)` constraint for both fresh installs and existing databases. Keyless events remain repeatable for intentionally unkeyed tracking. `src/Actions/RecordGoalEventAction.php`, `database/migrations/2026_05_31_000006_create_experiment_goal_events_table.php`, `database/migrations/2026_06_07_000001_add_idempotency_unique_to_experiment_goal_events_table.php`. (M)
- **Fix `value_amount` type drift** — `ExperimentGoalEventData::$valueAmount` is `?string` while the column and `ExperimentGoal::$value_amount` are `decimal(12,2)`. Coerce/validate to a numeric type to avoid silent string-into-decimal casts in reporting sums. `src/Data/ExperimentGoalEventData.php`. (S)
- **Shipped 2026-06-07: README + CHANGELOG integration docs.** The package now has a root README covering runtime variant resolution, goal-event recording, public-output boundaries, cross-package integration ownership, and focused verification. The changelog records the documentation addition alongside the existing health, allocation, significance, and cache-safety slices. `README.md`, `CHANGELOG.md`. (S)
- **Shipped 2026-06-07: Bounded candidate scan.** `ResolveExperimentVariantForContextAction` now limits request-context resolution with `capell-experiments.resolution_candidate_limit`, prefilters to experiments with active weighted variants, and eager-loads those variants so `AllocateVariantAction` can avoid a per-candidate variant query. `src/Actions/ResolveExperimentVariantForContextAction.php`, `src/Actions/AllocateVariantAction.php`, `config/capell-experiments.php`, `tests/Feature/ExperimentFoundationTest.php`. (M)

## 3. Missing Features (gaps)

Mapped against `capabilities[]` and A/B-testing norms:

- **Shipped 2026-06-04: Statistical significance (table-stakes).** `BuildWinnerReportAction` now reports baseline/control status, sample-size readiness, lift, p-value, confidence level, and overall significance. `DeclareExperimentWinnerAction` only persists a winner once the report has a statistically significant winning variant; a 1 allocation / 1 conversion raw leader remains undeclarable. `src/Actions/BuildWinnerReportAction.php`, `src/Data/WinnerReportData.php`, `src/Data/WinnerVariantReportData.php`, `tests/Feature/ExperimentFoundationTest.php`. (Differentiator.)
- **Owned frontend render surface remains future depth.** `ResolveExperimentVariantForContextAction` records frontend cache-safety metadata whenever it resolves a variant, but the package correctly keeps `capell.json` `surfaces` admin-only until it ships a route, view, Blade directive, Livewire component, middleware, or render hook. Assignment/resolution/goal recording remains reachable by consuming packages such as Campaign Studio calling the Actions. (Future differentiator.)
- **Shipped 2026-06-07: Scheduled → Active / auto-end automation.** `SyncExperimentStatusesAction` now transitions due scheduled experiments to active and expired scheduled/active experiments to ended. `capell:experiments:sync-statuses` runs the Action and the provider schedules it every five minutes with overlap protection. `src/Actions/SyncExperimentStatusesAction.php`, `src/Console/Commands/SyncExperimentStatusesCommand.php`, `src/Providers/ExperimentsServiceProvider.php`. (Table-stakes.)
- **Targeting depth** — `ExperimentContextData` exposes `path/query/utm/referrer/attributes/segments` but no `ipAddress`/`userAgent`, so no geo (core ships `torann/geoip`) or device/bot targeting, and no percentage-of-segment holdouts. (Differentiator.)
- **Mutually-exclusive experiments / exclusion groups** — A visitor can be allocated into every overlapping active experiment independently; there is no exclusion-group concept to prevent interaction effects. (Differentiator.)
- **Shipped 2026-06-07: Results dashboard.** Experiment records now expose a Filament results page and row/edit action backed by `BuildWinnerReportAction`, showing allocation totals, conversions, conversion rates, lift, p-values, sample readiness, and winner status in-product. `src/Filament/Resources/Experiments/Pages/ExperimentResultsPage.php`, `resources/views/filament/experiments/results.blade.php`, `src/Filament/Resources/Experiments/ExperimentResource.php`. (Table-stakes for a premium tier.)
- **Shipped 2026-06-08: Insights goal-event bridge.** `RecordGoalEventAction` now mirrors newly-created keyed goal events into Insights conversion events named `experiment.{experiment_key}.{goal_key}` when an Insights visit exists, giving Experiments a measurable cross-sell path without weakening package boundaries. (Table-stakes for the Growth bundle.)
- **Multi-goal / revenue reporting** — `value_amount` is captured per goal event but `BuildWinnerReportAction` only counts events; no revenue-per-variant or multi-goal funnel rollup despite the `experiment_goal_events_rollup_index`. (Differentiator.)

## 4. Issues / Risks

- **Shipped 2026-06-05: Static HTML cache no longer ignores resolved variant metadata.** Capell's frontend cache middleware (`frontend.cache`) serves static HTML on hit, which would freeze the first visitor's variant for everyone if variant resolution happened during public rendering without a contribution record. `ResolveExperimentVariantForContextAction` now records a frontend render contribution with `cacheable=false`, `sensitiveOutput=false`, cache tags, and `variesBy=["visitor","experiment","variant"]`; HTML Cache already consumes these records through `ExtensionCacheSafetyResolver`, so pages that resolve variants during render are blocked from shared static cache writes. Remaining gap: first-party variant-injection/goal-beacon surfaces.
- **Frontend allocation surface remains shallow.** The resolver now records cache-safety metadata for public renders, but every runtime allocation/resolution/goal path is still primarily exercised by package tests and consumer Actions. The capabilities `request-context-variant-resolution`, `personalization-payloads`, `cache-variation-metadata` are real code but have no package-owned Blade/Livewire/render-hook caller — keep the package admin-only in manifest metadata until that surface ships.
- **Shipped 2026-06-03: Stub health check contradicts its own manifest label.** `ExperimentsHealthCheck` now exposes `runDiagnostics()` / `passed()` and probes required experiment storage tables plus model table resolution. Tests cover both the healthy path and a missing required table. `src/Health/ExperimentsHealthCheck.php`, `tests/Feature/ExperimentsHealthCheckTest.php`.
- **Assignment determinism is sound but salt-collisions across experiments share the visitor's bucket position.** `stableNumber` namespaces by experiment id (`traffic:%d:%s`, `variant:%d:%s`) so cross-experiment correlation is avoided — good. However `traffic_percentage` gating uses `traffic:` hash while variant choice uses `variant:` hash; a visitor just inside the traffic threshold is fine, but there's no test pinning bucket stability across deploys/PHP versions (relies on `hash('sha256')` + `hexdec(substr(...,0,8))`). Add a golden-vector test. `src/Actions/AllocateVariantAction.php`.
- **Test gaps.** Only two feature files (`ExperimentFoundationTest`, `ExperimentAdminSurfaceTest`). Missing: anonymous/non-admin public-output safety (Capell core requirement — none, because there's no public output yet), cache-vary safety, statistical-significance behaviour, scheduled→active transition, audience optional-rule OR semantics edge cases, traffic-percentage exclusion, no Architecture suite asserting Actions use `AsAction`/`AsFake`.
- **Performance budget unverified.** Manifest sets `frontendRenderBudgetMs:20` / `adminQueryBudget:40` but the candidate-scan loop (§2) and per-experiment allocation lookups have no benchmark or query-count assertion. Cite `manifest.performance`.
- **i18n.** Strings are translated via `capell-experiments::generic` and enums use `getLabel()` — good. Only `en` is shipped; no other locales, acceptable for now but note for marketplace localisation.

## 5. Marketplace & Selling

**Current `summary`:** _"Run A/B tests with audience rules, sticky variant allocation, goal tracking, and winner reports."_ — accurate but lists internals (operators won't buy on "sticky variant allocation") and over-claims "winner reports" given there's no significance. **Composer `description`:** _"First-party experiments, A/B testing, variant allocation, personalization rules, and winner reporting for Capell."_ — diverges from the manifest description (manifest says "audience rules", composer says "personalization rules"); align the wording.

- **Improved 1-sentence summary:** "Run statistically-sound A/B tests on any Capell page or campaign — audience targeting, sticky bucketing, and goal-tracked winner reports, all server-side and cache-safe."
- **Improved 3–4 sentence description:** "Experiments brings native A/B and multivariate testing to Capell without third-party scripts or client-side flicker. Define experiments against pages or campaigns, target by path, UTM, referrer, query, or custom segments, and let deterministic sticky bucketing keep every visitor on the same variant across requests. Goals capture conversions and revenue, and winner reports surface lift with confidence so you ship the variant that actually won. Server-side allocation integrates with Capell's HTML cache and CDN so tests stay fast and fully cacheable." _(Note: the cache-safe claims remain aspirational until the frontend/cache integration row lands; significance is now shipped.)_
- **Media status:** the manifest promotes four runner-backed Capell admin screenshots covering experiment list, edit form, audience rules, and results/winner reporting. A short allocation-to-conversion-to-winner GIF remains optional future media polish.
- **Pricing/tier/bundle:** Premium tier in the `growth` bundle is now defensible for the current admin-first scope: operators get audience targeting, sticky or fresh weighted allocation, statistical winner reports, scheduled lifecycle automation, Campaign Studio consumption, and Insights conversion mirroring.
- **Cross-sell (declared `supports`):** `capell-app/insights` is now real through the goal-event conversion bridge; `capell-app/campaign-studio` syncs campaign variants/goals into experiments and reads result reports; `capell-app/html-cache` consumes the cache-vary contribution emitted during variant resolution; `capell-app/form-builder` can call the public goal-event Action from form-submission listeners.
- **Differentiators / value props / target buyer:** No-flicker server-side testing inside the CMS; cache-compatible; first-party (no Optimizely/VWO subscription). Buyer = growth/marketing operators on Capell sites who want CRO without bolting on an external experimentation SaaS.
- **Keywords/tags:** ab-testing, split-testing, experimentation, conversion-optimization, cro, variant-allocation, audience-targeting, personalization, goal-tracking, statistical-significance, server-side-testing, cache-safe-experiments.

## Completion Review

Completed 2026-06-08. The current manifest-backed plan is closed: Experiments ships admin resources, result reporting, deterministic and configurable allocation strategies, cache-safety metadata, scheduled status automation, idempotent keyed goal events, statistical significance gates, diagnostics, runner-backed Marketplace screenshots, Campaign Studio consumption, and Insights conversion mirroring. A first-party frontend render/beacon surface, geo/device targeting, exclusion groups, revenue rollups, and golden-vector/architecture depth remain future product candidates.

## 6. Prioritized Roadmap

| Item                                                                                                 | Bucket | Effort | Impact | Section ref |
| ---------------------------------------------------------------------------------------------------- | ------ | ------ | ------ | ----------- |
| Shipped 2026-06-05: Cache-safe resolver integration (vary key → html-cache) + safety coverage        | Done   | L      | High   | §3, §4      |
| Shipped 2026-06-04: Add statistical significance + sample-size floor to winner reporting/declaration | Done   | M      | High   | §3          |
| Shipped 2026-06-03: Implement real `ExperimentsHealthCheck` probes                                   | Done   | S      | High   | §4          |
| Shipped 2026-06-03: Filter `candidateQuery` by `subject_class`                                       | Done   | S      | High   | §2          |
| Shipped 2026-06-04: Honour `allocation_strategy`                                                     | Done   | M      | Med    | §2          |
| Shipped 2026-06-07: Goal-event idempotency guard + unique key                                        | Done   | M      | High   | §2, §3      |
| Shipped 2026-06-07: Scheduled→Active / auto-end command + scheduler                                  | Done   | M      | Med    | §3          |
| Shipped 2026-06-07: Results dashboard (Filament page/widget)                                         | Done   | L      | High   | §3, §5      |
| Shipped 2026-06-07: Bounded candidate scan + query-count benchmark vs budget                         | Done   | M      | Med    | §2, §4      |
| Shipped 2026-06-07: Add README + CHANGELOG + integration docs                                        | Done   | S      | Med    | §2, §5      |
| Shipped 2026-06-07: Capture runner-backed admin screenshots for marketplace listing                  | Done   | S      | Med    | §5          |
| Shipped 2026-06-08: Insights + Campaign Studio cross-sell integrations                               | Done   | L      | Med    | §5          |
| First-party frontend variant injection and goal beacon surface                                       | Future | L      | High   | §3, §4      |
| Geo/device targeting via `ipAddress`/`userAgent` + geoip                                             | Future | M      | Med    | §3          |
| Mutually-exclusive experiment groups                                                                 | Future | L      | Med    | §3          |
| Revenue/multi-goal rollup reporting                                                                  | Future | M      | Med    | §3          |
| Golden-vector determinism test + Architecture suite                                                  | Future | S      | Med    | §4          |