Experiments — Improvement & Growth Plan

Package: capell-app/experiments · Kind: package · Tier: premium · Product group: Capell Growth · Bundle: growth · Status: Complete

1. Snapshot

Experiments is a server-side A/B testing engine for Capell built entirely from Laravel Actions over six tables (experiments, experiment_variants, experiment_goals, experiment_audience_rules, experiment_allocations, experiment_goal_events). Domain logic lives in src/Actions: CreateExperimentAction (aggregate writer), AllocateVariantAction (deterministic SHA-256 weighted bucketing + sticky allocation row), ResolveExperimentVariantForContextAction (candidate query → allocate → ResolvedExperimentVariantData with cache-vary metadata), EvaluateAudienceRulesAction (path/query/utm/referrer/attribute/segment operators), RecordGoalEventAction, BuildWinnerReportAction, and DeclareExperimentWinnerAction. Surface delivery is admin-only: four Filament resources (Experiment/Variant/Goal/AudienceRule) plus the result page contributed via ExperimentsServiceProvider::registerAdminResources(); depends on capell-app/admin, capell-app/core, Filament, spatie/laravel-data, lorisleiva/laravel-actions. Current marketplace summary verbatim: “Run A/B tests with audience rules, sticky variant allocation, goal tracking, and winner reports.” The manifest now promotes four runner-backed Capell admin screenshots for list/edit/rules/results surfaces.

Completed Improvement Slices

• 2026-06-03: Implemented real ExperimentsHealthCheck diagnostics and filtered request-context candidate resolution by subject_class.
• 2026-06-04: Honoured allocation_strategy in AllocateVariantAction, added allocation ids to allocation data, gated winner reports/declarations behind a minimum sample size plus two-proportion significance check, and reconciled manifest/Composer copy with the shipped admin-only surface and statistical-significance capability.
• 2026-06-05: Made ResolveExperimentVariantForContextAction record a non-cacheable frontend render contribution with visitor/experiment/variant vary metadata, so HTML Cache will not persist one visitor’s resolved variant as shared anonymous HTML.
• 2026-06-08: Connected keyed experiment goal events into Insights conversion reporting when the allocation carries an Insights visit uuid, preserving idempotency and source-package metadata for Growth bundle dashboards.

2. Improvements (existing functionality)

• Shipped 2026-06-04: Honour allocation_strategy or delete it. AllocateVariantAction now branches on AllocationStrategy: StickyWeighted reuses a persisted allocation row for the visitor hash, while Weighted records a fresh weighted allocation without using sticky lookup state. Allocation data now includes the persisted allocation id so downstream goal/event callers can keep an explicit Action/Data boundary. src/Actions/AllocateVariantAction.php, src/Data/VariantAllocationData.php, tests/Feature/ExperimentFoundationTest.php. (M)
• Shipped 2026-06-03: Filter candidateQuery by subject_class. ResolveExperimentVariantForContextAction::candidateQuery() now filters by subject_class while preserving class-agnostic experiments, preventing collisions between experiments that share subject_type + subject_id across different subject classes. src/Actions/ResolveExperimentVariantForContextAction.php, tests/Feature/ResolveVariantSubjectClassTest.php. (S)
• Shipped 2026-06-07: Goal-event idempotency guard. RecordGoalEventAction now uses firstOrCreate when callers provide an event_key, backed by a unique (experiment_allocation_id, experiment_goal_id, event_key) constraint for both fresh installs and existing databases. Keyless events remain repeatable for intentionally unkeyed tracking. src/Actions/RecordGoalEventAction.php, database/migrations/2026_05_31_000006_create_experiment_goal_events_table.php, database/migrations/2026_06_07_000001_add_idempotency_unique_to_experiment_goal_events_table.php. (M)
• Fix value_amount type drift — ExperimentGoalEventData::$valueAmount is ?string while the column and ExperimentGoal::$value_amount are decimal(12,2). Coerce/validate to a numeric type to avoid silent string-into-decimal casts in reporting sums. src/Data/ExperimentGoalEventData.php. (S)
• Shipped 2026-06-07: README + CHANGELOG integration docs. The package now has a root README covering runtime variant resolution, goal-event recording, public-output boundaries, cross-package integration ownership, and focused verification. The changelog records the documentation addition alongside the existing health, allocation, significance, and cache-safety slices. README.md, CHANGELOG.md. (S)
• Shipped 2026-06-07: Bounded candidate scan. ResolveExperimentVariantForContextAction now limits request-context resolution with capell-experiments.resolution_candidate_limit, prefilters to experiments with active weighted variants, and eager-loads those variants so AllocateVariantAction can avoid a per-candidate variant query. src/Actions/ResolveExperimentVariantForContextAction.php, src/Actions/AllocateVariantAction.php, config/capell-experiments.php, tests/Feature/ExperimentFoundationTest.php. (M)

3. Missing Features (gaps)

Mapped against capabilities[] and A/B-testing norms:

• Shipped 2026-06-04: Statistical significance (table-stakes). BuildWinnerReportAction now reports baseline/control status, sample-size readiness, lift, p-value, confidence level, and overall significance. DeclareExperimentWinnerAction only persists a winner once the report has a statistically significant winning variant; a 1 allocation / 1 conversion raw leader remains undeclarable. src/Actions/BuildWinnerReportAction.php, src/Data/WinnerReportData.php, src/Data/WinnerVariantReportData.php, tests/Feature/ExperimentFoundationTest.php. (Differentiator.)
• Owned frontend render surface remains future depth. ResolveExperimentVariantForContextAction records frontend cache-safety metadata whenever it resolves a variant, but the package correctly keeps capell.json surfaces admin-only until it ships a route, view, Blade directive, Livewire component, middleware, or render hook. Assignment/resolution/goal recording remains reachable by consuming packages such as Campaign Studio calling the Actions. (Future differentiator.)
• Shipped 2026-06-07: Scheduled → Active / auto-end automation. SyncExperimentStatusesAction now transitions due scheduled experiments to active and expired scheduled/active experiments to ended. capell:experiments:sync-statuses runs the Action and the provider schedules it every five minutes with overlap protection. src/Actions/SyncExperimentStatusesAction.php, src/Console/Commands/SyncExperimentStatusesCommand.php, src/Providers/ExperimentsServiceProvider.php. (Table-stakes.)
• Targeting depth — ExperimentContextData exposes path/query/utm/referrer/attributes/segments but no ipAddress/userAgent, so no geo (core ships torann/geoip) or device/bot targeting, and no percentage-of-segment holdouts. (Differentiator.)
• Mutually-exclusive experiments / exclusion groups — A visitor can be allocated into every overlapping active experiment independently; there is no exclusion-group concept to prevent interaction effects. (Differentiator.)
• Shipped 2026-06-07: Results dashboard. Experiment records now expose a Filament results page and row/edit action backed by BuildWinnerReportAction, showing allocation totals, conversions, conversion rates, lift, p-values, sample readiness, and winner status in-product. src/Filament/Resources/Experiments/Pages/ExperimentResultsPage.php, resources/views/filament/experiments/results.blade.php, src/Filament/Resources/Experiments/ExperimentResource.php. (Table-stakes for a premium tier.)
• Shipped 2026-06-08: Insights goal-event bridge. RecordGoalEventAction now mirrors newly-created keyed goal events into Insights conversion events named experiment.{experiment_key}.{goal_key} when an Insights visit exists, giving Experiments a measurable cross-sell path without weakening package boundaries. (Table-stakes for the Growth bundle.)
• Multi-goal / revenue reporting — value_amount is captured per goal event but BuildWinnerReportAction only counts events; no revenue-per-variant or multi-goal funnel rollup despite the experiment_goal_events_rollup_index. (Differentiator.)

4. Issues / Risks

• Shipped 2026-06-05: Static HTML cache no longer ignores resolved variant metadata. Capell’s frontend cache middleware (frontend.cache) serves static HTML on hit, which would freeze the first visitor’s variant for everyone if variant resolution happened during public rendering without a contribution record. ResolveExperimentVariantForContextAction now records a frontend render contribution with cacheable=false, sensitiveOutput=false, cache tags, and variesBy=["visitor","experiment","variant"]; HTML Cache already consumes these records through ExtensionCacheSafetyResolver, so pages that resolve variants during render are blocked from shared static cache writes. Remaining gap: first-party variant-injection/goal-beacon surfaces.
• Frontend allocation surface remains shallow. The resolver now records cache-safety metadata for public renders, but every runtime allocation/resolution/goal path is still primarily exercised by package tests and consumer Actions. The capabilities request-context-variant-resolution, personalization-payloads, cache-variation-metadata are real code but have no package-owned Blade/Livewire/render-hook caller — keep the package admin-only in manifest metadata until that surface ships.
• Shipped 2026-06-03: Stub health check contradicts its own manifest label. ExperimentsHealthCheck now exposes runDiagnostics() / passed() and probes required experiment storage tables plus model table resolution. Tests cover both the healthy path and a missing required table. src/Health/ExperimentsHealthCheck.php, tests/Feature/ExperimentsHealthCheckTest.php.
• Assignment determinism is sound but salt-collisions across experiments share the visitor’s bucket position. stableNumber namespaces by experiment id (traffic:%d:%s, variant:%d:%s) so cross-experiment correlation is avoided — good. However traffic_percentage gating uses traffic: hash while variant choice uses variant: hash; a visitor just inside the traffic threshold is fine, but there’s no test pinning bucket stability across deploys/PHP versions (relies on hash('sha256') + hexdec(substr(...,0,8))). Add a golden-vector test. src/Actions/AllocateVariantAction.php.
• Test gaps. Only two feature files (ExperimentFoundationTest, ExperimentAdminSurfaceTest). Missing: anonymous/non-admin public-output safety (Capell core requirement — none, because there’s no public output yet), cache-vary safety, statistical-significance behaviour, scheduled→active transition, audience optional-rule OR semantics edge cases, traffic-percentage exclusion, no Architecture suite asserting Actions use AsAction/AsFake.
• Performance budget unverified. Manifest sets frontendRenderBudgetMs:20 / adminQueryBudget:40 but the candidate-scan loop (§2) and per-experiment allocation lookups have no benchmark or query-count assertion. Cite manifest.performance.
• i18n. Strings are translated via capell-experiments::generic and enums use getLabel() — good. Only en is shipped; no other locales, acceptable for now but note for marketplace localisation.

5. Marketplace & Selling

Current summary: “Run A/B tests with audience rules, sticky variant allocation, goal tracking, and winner reports.” — accurate but lists internals (operators won’t buy on “sticky variant allocation”) and over-claims “winner reports” given there’s no significance. Composer description: “First-party experiments, A/B testing, variant allocation, personalization rules, and winner reporting for Capell.” — diverges from the manifest description (manifest says “audience rules”, composer says “personalization rules”); align the wording.

• Improved 1-sentence summary: “Run statistically-sound A/B tests on any Capell page or campaign — audience targeting, sticky bucketing, and goal-tracked winner reports, all server-side and cache-safe.”
• Improved 3–4 sentence description: “Experiments brings native A/B and multivariate testing to Capell without third-party scripts or client-side flicker. Define experiments against pages or campaigns, target by path, UTM, referrer, query, or custom segments, and let deterministic sticky bucketing keep every visitor on the same variant across requests. Goals capture conversions and revenue, and winner reports surface lift with confidence so you ship the variant that actually won. Server-side allocation integrates with Capell’s HTML cache and CDN so tests stay fast and fully cacheable.” (Note: the cache-safe claims remain aspirational until the frontend/cache integration row lands; significance is now shipped.)
• Media status: the manifest promotes four runner-backed Capell admin screenshots covering experiment list, edit form, audience rules, and results/winner reporting. A short allocation-to-conversion-to-winner GIF remains optional future media polish.
• Pricing/tier/bundle: Premium tier in the growth bundle is now defensible for the current admin-first scope: operators get audience targeting, sticky or fresh weighted allocation, statistical winner reports, scheduled lifecycle automation, Campaign Studio consumption, and Insights conversion mirroring.
• Cross-sell (declared supports): capell-app/insights is now real through the goal-event conversion bridge; capell-app/campaign-studio syncs campaign variants/goals into experiments and reads result reports; capell-app/html-cache consumes the cache-vary contribution emitted during variant resolution; capell-app/form-builder can call the public goal-event Action from form-submission listeners.
• Differentiators / value props / target buyer: No-flicker server-side testing inside the CMS; cache-compatible; first-party (no Optimizely/VWO subscription). Buyer = growth/marketing operators on Capell sites who want CRO without bolting on an external experimentation SaaS.
• Keywords/tags: ab-testing, split-testing, experimentation, conversion-optimization, cro, variant-allocation, audience-targeting, personalization, goal-tracking, statistical-significance, server-side-testing, cache-safe-experiments.

Completion Review

Completed 2026-06-08. The current manifest-backed plan is closed: Experiments ships admin resources, result reporting, deterministic and configurable allocation strategies, cache-safety metadata, scheduled status automation, idempotent keyed goal events, statistical significance gates, diagnostics, runner-backed Marketplace screenshots, Campaign Studio consumption, and Insights conversion mirroring. A first-party frontend render/beacon surface, geo/device targeting, exclusion groups, revenue rollups, and golden-vector/architecture depth remain future product candidates.

6. Prioritized Roadmap

Item	Bucket	Effort	Impact	Section ref
Shipped 2026-06-05: Cache-safe resolver integration (vary key → html-cache) + safety coverage	Done	L	High	§3, §4
Shipped 2026-06-04: Add statistical significance + sample-size floor to winner reporting/declaration	Done	M	High	§3
Shipped 2026-06-03: Implement real ExperimentsHealthCheck probes	Done	S	High	§4
Shipped 2026-06-03: Filter candidateQuery by subject_class	Done	S	High	§2
Shipped 2026-06-04: Honour allocation_strategy	Done	M	Med	§2
Shipped 2026-06-07: Goal-event idempotency guard + unique key	Done	M	High	§2, §3
Shipped 2026-06-07: Scheduled→Active / auto-end command + scheduler	Done	M	Med	§3
Shipped 2026-06-07: Results dashboard (Filament page/widget)	Done	L	High	§3, §5
Shipped 2026-06-07: Bounded candidate scan + query-count benchmark vs budget	Done	M	Med	§2, §4
Shipped 2026-06-07: Add README + CHANGELOG + integration docs	Done	S	Med	§2, §5
Shipped 2026-06-07: Capture runner-backed admin screenshots for marketplace listing	Done	S	Med	§5
Shipped 2026-06-08: Insights + Campaign Studio cross-sell integrations	Done	L	Med	§5
First-party frontend variant injection and goal beacon surface	Future	L	High	§3, §4
Geo/device targeting via ipAddress/userAgent + geoip	Future	M	Med	§3
Mutually-exclusive experiment groups	Future	L	Med	§3
Revenue/multi-goal rollup reporting	Future	M	Med	§3
Golden-vector determinism test + Architecture suite	Future	S	Med	§4