feat(telemetry): fix creation-funnel tracking gaps (#47386)

## Summary

The creation-funnel instrumentation that shipped Jun 25 (#47291, #47293)
had real gaps, surfaced by the weekly telemetry audit and confirmed
against production PostHog data before I touched code. The two automated
reports also contradicted each other on `errorReason`; I checked
production (every value is a controlled slug) and the emit path (only
`useTrackFunnelError` sets it, and it only accepts classified slugs), so
I left the type as-is rather than add a cross-package abstraction for a
risk that cannot occur today.

## Changes

- Classify HTTP 401/403/404 API errors as `unauthorized` / `forbidden` /
`not_found` instead of the catch-all `other`. In production the
`org_creation` `other` bucket was ~96% 401s (~1,300 real over 4 days),
invisible in reason breakdowns. The status-code fallback runs after the
message-pattern match, so specific reasons still win and it only rescues
errors that would otherwise be `other`.
- Add a single `tier` property (`tier_free` / `tier_pro` / `tier_payg` /
`tier_team`) to `organization_creation_completed`, which previously
carried no properties. One canonical billing slug (matching
`SubscriptionTier`) instead of two overlapping plan/tier fields, so the
org-creation funnel segments cleanly by tier and joins against
subscription data. `tier_payg` is uncapped PRO.
- Freeze the submitted tier at submit time (snapshot in `createOrg`)
rather than reading live form state in the success callback, so the
event records the tier that was actually created even if the user edits
the form during the async payment flow.
- Emit `project_creation_form_exposed` with `surface: 'vercel'` on the
integration deploy-button project-creation page (the enum value existed
but was never fired). Gated on the URL `slug` so the impression is
captured as soon as the form renders, matching the sibling exposure hook
on that page.

I also checked the confirm-modal error path flagged in the insights
post: it already classifies via the shared
`useProjectCreateMutation.onError`, so adding instrumentation there
would double-count. No change made.

## Testing

These are analytics events with no UI change, so correctness is in what
lands in PostHog. Post-deploy validation I will run against production
(project 34344):

- `dashboard_error_created` where `origin='org_creation'` and
`errorReason='other'` drops ~96%, with `unauthorized` / `not_found`
appearing.
- `organization_creation_completed.tier` populated on 100% of new events
with one of the four tier slugs.
- `project_creation_form_exposed` with `surface='vercel'` goes from 0 to
greater than 0.

## Linear

- fixes GROWTH-948


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added telemetry for organization creation completion that includes the
selected billing tier.
* Added one-time telemetry when the Vercel project creation form is
exposed.
* **Bug Fixes**
* Improved API error classification to more accurately distinguish
unauthorized, forbidden, and not found responses.
* **Documentation**
* Updated telemetry event definitions to require tier metadata for
organization creation events.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Pamela Chia authored and GitHub committed 2026-06-30 23:11:53 +08:00
1 parent d7543d536f
commit 98cfe3307e
4 files changed
+51 -7

No files matched your search

@@ -298,7 +298,13 @@ export const NewOrgForm = ({
}
const onOrganizationCreated = (org: { slug: string }) => {
track('organization_creation_completed', undefined, { organization: org.slug })
if (submittedTier.current) {
track(
'organization_creation_completed',
{ tier: submittedTier.current },
{ organization: org.slug }
)
}
const prefilledProjectName = user.profile?.username
? user.profile.username + `'s Project`
@@ -326,6 +332,8 @@ export const NewOrgForm = ({
} as StripeElementsOptions
}, [paymentIntentSecret, resolvedTheme])
const submittedTier = useRef<'tier_free' | 'tier_pro' | 'tier_payg' | 'tier_team' | null>(null)
async function createOrg(
formValues: z.infer<typeof formSchema>,
paymentMethodId?: string,
@@ -336,15 +344,17 @@ export const NewOrgForm = ({
}
) {
const dbTier = formValues.plan === 'PRO' && !formValues.spend_cap ? 'PAYG' : formValues.plan
const tier = ('tier_' + dbTier.toLowerCase()) as
| 'tier_payg'
| 'tier_pro'
| 'tier_free'
| 'tier_team'
submittedTier.current = tier
createOrganization({
name: formValues.name,
kind: formValues.kind,
tier: ('tier_' + dbTier.toLowerCase()) as
| 'tier_payg'
| 'tier_pro'
| 'tier_free'
| 'tier_team',
tier,
...(formValues.kind == 'COMPANY' ? { size: formValues.size } : {}),
payment_method: paymentMethodId,
billing_name: dbTier === 'FREE' ? undefined : customerData?.billing_name,
@@ -74,6 +74,9 @@ const GENERIC_REASONS = [
'payment_failed',
'payment_error',
'oriole_unavailable',
'unauthorized',
'forbidden',
'not_found',
'other',
] as const
@@ -85,6 +88,12 @@ export type FunnelErrorReason =
| ValuesOf<typeof STRIPE_DECLINE_REASONS>
| (typeof GENERIC_REASONS)[number]
const STATUS_REASONS: Readonly<Partial<Record<number, FunnelErrorReason>>> = {
401: 'unauthorized',
403: 'forbidden',
404: 'not_found',
}
export function classifyApiError(origin: FunnelOrigin, error: unknown): FunnelErrorClassification {
const err = error as { code?: unknown; errorType?: unknown; message?: unknown }
const code = typeof err?.code === 'number' ? err.code : undefined
@@ -107,6 +116,10 @@ export function classifyApiError(origin: FunnelOrigin, error: unknown): FunnelEr
return { errorCategory: 'api', errorReason: reason, errorCode: code }
}
}
const statusReason = STATUS_REASONS[code]
if (statusReason) {
return { errorCategory: 'api', errorReason: statusReason, errorCode: code }
}
return { errorCategory: 'api', errorReason: 'other', errorCode: code }
}
@@ -117,6 +117,14 @@ const CreateProject = () => {
const { data: organizationData } = useOrganizationsQuery()
const organization = organizationData?.find((x) => x.slug === slug)
const hasTrackedFormExposed = useRef(false)
useEffect(() => {
if (hasTrackedFormExposed.current) return
if (!slug) return
hasTrackedFormExposed.current = true
track('project_creation_form_exposed', { surface: 'vercel' }, { organization: slug })
}, [slug, track])
/**
* array of integrations installed
*/
+14 -1
View File
@@ -480,6 +480,12 @@ export interface OrganizationCreationFormExposedEvent {
*/
export interface OrganizationCreationCompletedEvent {
action: 'organization_creation_completed'
properties: {
/**
* Billing tier provisioned at creation. tier_payg is uncapped PRO.
*/
tier: 'tier_free' | 'tier_pro' | 'tier_payg' | 'tier_team'
}
groups: Omit<TelemetryGroups, 'project'>
}
@@ -2930,7 +2936,14 @@ export interface DashboardErrorCreatedEvent {
*/
errorCategory?: 'validation' | 'api' | 'network' | 'payment' | 'unknown'
/**
* Controlled-vocabulary slug describing the reason (no free text, no PII)
* Controlled-vocabulary slug describing the reason (no free text, no PII).
*
* Typed `string` rather than a literal union on purpose: the source-of-truth
* union `FunnelErrorReason` lives in `apps/studio/lib/telemetry/funnel-errors.ts`,
* and this `common` package cannot import from an app. The constraint is enforced
* at the only emit site instead: `useTrackFunnelError` accepts a classified
* `FunnelErrorReason`, so free text never reaches this field. Do not widen usage by
* setting `errorReason` from a raw error message.
*/
errorReason?: string
/**