Integrating ClickHouse with Circleback
The Circleback integration runs five independently selectable raw resource streams within one account pipeline.
This integration is part of chkit Sync (beta), the optional TypeScript ingestion layer built on chkit Schema.
Install
Section titled “Install”bunx chkit add circlebackbunx chkit checkbunx chkit generate --name add_circlebackbunx chkit migrate --applybunx chkit ingest run --tag provider:circlebackSet CIRCLEBACK_API_KEY, or pass apiKey in the configuration. Create a key through Circleback Settings → API keys; see the authentication reference. Native JSON requires ClickHouse 25.3 or later.
Edit src/integrations/circleback/config.ts for a stable sourceId, meeting ownership, and setup-time database. ownership: 'All' includes shared meetings accessible to the authenticated user. People, companies, and action items use the provider’s own authenticated-user collections; these endpoints do not promise a full organization export.
pipeline.ts composes the five resources, sources/ owns raw schemas and readers, and client.ts handles requests and Link pagination. The factory createCirclebackPipeline(config) binds a snapshot of reader configuration, including apiKey; each stream checks it when it runs, before any request. Runtime factories use existing exported schemas; storage changes belong before schema discovery and migration generation.
| Resource | Default ClickHouse table | Records synced | API reference |
|---|---|---|---|
Meetings (meetings) | circleback_meetings_raw | Accessible meeting provider objects and notes, with tag names as string arrays. | GET/meetings |
Meeting transcripts (meeting_transcripts) | circleback_meeting_transcripts_raw | Whole transcript snapshots per meeting and separate latest read outcomes; availability failures preserve earlier successful text. | GET/meetings, GET/meeting/{meetingId}/transcript |
Action items (action_items) | circleback_action_items_raw | Raw action items for any assignee, fully paginated across PENDING and DONE completion statuses. | GET/action-items |
People (people) | circleback_people_raw | Complete same-person listing and detail fields, preserving external references and native person/company IDs. | GET/people, GET/person/{profileId} |
Companies (companies) | circleback_companies_raw | Domain-keyed company listing and detail fields with a nullable, corroborated native numeric company ID for joins. | GET/companies, GET/company/{domain} |
Raw resource boundaries
Section titled “Raw resource boundaries”Meetings, meeting transcripts, action items, people, and companies have separate streams, tables, and journal completion. Each runs alone and owns any required discovery. The transcript reader lists its own eligible meetings; selecting transcripts does not depend on a prior meetings run, and transcript failure does not block raw meeting publication.
Raw provider fields remain under data with source_id metadata. Provider-returned meeting attendee/action-item summaries and recording URLs remain intact; readers do not assemble a meeting by fetching its child streams. The requested tag projection stores all returned tag names as string[], without a tag-definition stream. No calendar-event or binary-download reader is included.
People fetch one same-person detail per listed profile and validate the native ID, preserving externalLinks and all other returned fields. Companies likewise fetch one detail per domain. These same-resource requests add to full-read cost and require matching access.
bunx chkit ingest run --tag provider:circleback --tag resource:meeting_transcriptsbunx chkit ingest status --tag provider:circleback --jsonFull sync and recovery
Section titled “Full sync and recovery”Circleback exposes paginated full collections without a reliable modification feed. Every resource uses native fullSync() completion and paginate(). Each run exhausts its own provider pages; failed or interrupted reads restart from the first page, and empty successful reads still record completion. Progress never declares successful completion before destination acknowledgement. Loaded rows may remain visible after a later failure.
Link pagination preserves ownership, assignee and completion filters even on cursor-only continuations. Conflicting filters, repeated/unsafe links and malformed provider responses fail visibly. Requests go through the ingestion runtime, which forwards cancellation, times out after 30 seconds, and retries only transient failures; HTTP 429 retries honor Retry-After, falling back to Circleback’s RateLimit-Reset seconds when Retry-After is absent or not parsed by the shared HTTP parser. Blank and negative Retry-After values currently parse as zero and bypass the fallback. Without either parsed delay, normal executor backoff applies. This fallback is fixture-tested hardening, not evidence of a live provider incident.
Transcripts are rechecked for every listed meeting, including unchanged old meetings. A meeting’s updatedAt is not treated as transcript readiness. The action-item listing explicitly uses assigneeType=Anyone and separately exhausts PENDING and DONE, the native statuses verified in the official OpenAPI and update contract. Unexpected statuses fail instead of silently producing a partial export.
Full reads and status partitions are mutable, not atomic snapshots. A subsequent full run reconciles late availability, edits and status changes. Absent resources remain stored, and inaccessible transcripts do not prove deletion. Full-sync readers ignore date backfill bounds and provide no date-range backfill. Use an external scheduler; ChKit supports overlapping runs with independent journal histories.
Identities and joins
Section titled “Identities and joins”Row IDs are [sourceId, nativeId] for meetings, action items and people; companies always use [sourceId, domain]. Native provider IDs and references remain unchanged in data. Keep source_id in every cross-resource join.
Company responses preserve listing and same-company detail fields. The company_id envelope metadata preserves a native list/detail ID or a consistent non-null company ID from returned people. Conflicting numeric IDs fail. Empty companies without a numeric ID remain valid domain-keyed rows with company_id: null; no numeric mapping is invented. Join person data.companyId to company company_id only when known.
Account or filter changes require a new sourceId and deliberately configured destinations. Version 0.1.0 used direct meeting JSON and unscoped keys. The 0.2.0 source envelopes, scoped IDs, string tags and additional tables require deliberate migration or reingestion; earlier rows are not automatically rewritten.
Transcript content and read outcomes
Section titled “Transcript content and read outcomes”Each transcript has one stable full-content key [sourceId, meetingId] and one read-outcome key [sourceId, meetingId, 'availability']. The envelope discriminator is kind: 'transcript' or kind: 'availability'.
HTTP 200 writes the complete raw segment array and an available outcome. HTTP 403 or 404 updates only the outcome to forbidden or not_found, preserving earlier successful text. A successful empty array replaces the prior content with []. Malformed transcript data fails without marking availability successful. Segments have no synthetic IDs.
See destinations.
Select successful snapshots with raw.kind::String = 'transcript'; join the latest outcome separately:
WITH snapshots AS ( SELECT raw.source_id::String AS source_id, raw.meeting_id::String AS meeting_id, JSONExtractRaw(toJSONString(raw), 'data') AS transcript FROM default.circleback_meeting_transcripts_raw FINAL WHERE raw.kind::String = 'transcript'), outcomes AS ( SELECT raw.source_id::String AS source_id, raw.meeting_id::String AS meeting_id, raw.status::String AS latest_status FROM default.circleback_meeting_transcripts_raw FINAL WHERE raw.kind::String = 'availability')SELECT s.source_id, s.meeting_id, s.transcript, o.latest_statusFROM snapshots AS sLEFT JOIN outcomes AS o ON s.source_id = o.source_id AND s.meeting_id = o.meeting_id;An unavailable latest outcome can coexist with a retained successful snapshot. Neither unavailable status is assumed to mean temporary processing or deletion.
Test the readers
Section titled “Test the readers”bunx chkit add circleback --with-testsbun test src/integrations/circleback/tests/basic.test.tsThe installed fixture suite requires no credentials or live database and verifies independently selected resources, complete status partitions, source/sink replay, late transcripts, retained content, native joins and pagination failures.
Changelog
Section titled “Changelog”Version 0.3.1
- Trim raw-storage notes from the README.
Version 0.3.0
- Read configuration and the API key when a stream runs, so `generate`, `migrate`, and `chkit check` never depend on them; invalid settings or a missing API key fail that run before any request. Requires zod 4 and chkit 0.2.0-beta.12.
- Send requests through the ingestion runtime, which retries only network failures, timeouts, retryable HTTP statuses, and HTTP 429 rate limits; malformed payloads and invalid Link continuations fail at once.
- Set `apiKey` in the configuration to use another API key, for example for a second installation; it defaults to `CIRCLEBACK_API_KEY`.
Version 0.2.1
- Add fixture-tested RateLimit-Reset seconds fallback on HTTP 429 when Retry-After is absent or not parsed by the shared HTTP parser, without changing permission failures or transcript availability handling.
- Clarify native JSON normalization of explicit null paths and empty objects, including nullable transcript speakers.
Version 0.2.0
- Provide one configurable account pipeline with independent meetings, meeting transcripts, action items, people, and companies raw streams.
- Use native full-sync completion and safe first-page replay with scoped identities and validated Link pagination that preserves query filters.
- Revisit every eligible meeting for late transcripts; retain successful snapshots separately from forbidden/not-found read outcomes.
- Read action items for anyone across both native PENDING and DONE statuses; preserve people and company relationships without inventing numeric company IDs.
- Store complete tag names as string arrays and raw provider payloads in source-scoped envelopes; adopting this layout requires reingestion or migration of earlier unscoped meeting observations.
- Add portable fixtures for independently selected resources, reversed stream order, malformed responses, pagination, late transcripts, and source/destination recovery.
- Consume full pagination pages and use shared continuation-cycle detection while preserving independent resource checkpoints and provider Link validation.
Version 0.1.0
- Introduce raw meeting ingestion with transcript enrichment.