Bring Your Own Storage (BYOS)

BYOS lets your organization store reconciliation data artifacts — the parquet files and analysis outputs produced by your reconciliation jobs — in a GCS bucket you own and control, rather than DataRecs’ platform-managed storage.

Overview

Data sovereignty

Your reconciliation data lives in your own GCS bucket, in your own GCP project. DataRecs accesses it only when running your reconciliation jobs.

Zero-downtime migration

The migration copies your data while reads continue uninterrupted. Writes are paused only during the active copy window (typically minutes).

Fully reversible

Changed your mind? Revert to platform-managed storage at any time — the same pipeline copies data back.

Verified integrity

Every copied object is verified by count, size, and CRC32C checksum before the migration is marked complete.

How it works

The migration proceeds through a series of automated phases. You trigger it with a single API call or Console click, and it runs to completion without further intervention.

Migration phases

Phase	Duration	What happens
Lock acquisition	Instant	Exclusive lock prevents concurrent migrations on your tenant.
Write pause	Instant	New data writes are temporarily paused (reads continue). Scheduled reconciliation jobs are suspended.
Drain	0–60s	Any currently-running reconciliation job is allowed to finish (or gracefully stopped if it exceeds the deadline).
Copy	Minutes	Data artifacts are streamed from platform storage to your GCS bucket (or vice versa for a revert).
Verify	Seconds	Object counts, total sizes, and per-file checksums are compared to confirm integrity.
Finalize	Instant	Your storage mode is switched, scheduled jobs resume, and the write pause is lifted.

Note

The total migration time depends on the volume of data. A tenant with a few hundred parquet files typically migrates in under 5 minutes. Large tenants with thousands of files may take up to an hour.

What gets migrated

Only reconciliation data artifacts are migrated:

• Parquet data files (your actual reconciliation results)
• Drill-down filter files (used for targeted re-extraction)

Workflow logs, system metadata, and intermediate processing files are not migrated — they remain in platform storage and do not affect your reconciliation results.

Artifact encryption posture

Where your artifacts live (platform storage vs. your own bucket) and how they are encrypted are two separate choices. The artifact encryption posture lets you pick from four combinations, trading off convenience against how much you can read your own data — and how zero-knowledge it is to DataRecs.

Posture	Where it’s stored	How it’s encrypted	Can you read it?
Platform managed (default)	DataRecs platform bucket	A key DataRecs manages for you	No
Your bucket, our key	Your GCS bucket	A key DataRecs manages for you	No
Your bucket, your key (BYOK)	Your GCS bucket	Your own Cloud KMS key	Yes
Your bucket, no encryption	Your GCS bucket	Not encrypted by DataRecs	Yes

The readability vs. zero-knowledge trade-off

• Platform managed and Your bucket, our key are zero-knowledge to you: the artifacts are encrypted with a key only DataRecs holds, so even though the second option puts the files in your own bucket, the contents stay opaque to you. Choose these when you don’t need to read the raw artifacts yourself and want the simplest setup.
• Your bucket, your key (BYOK) is encrypted yet readable: you control the KMS key, so only you (and DataRecs, while you grant access) can decrypt — and you can revoke access at any time. This is the strongest option for organizations that want both encryption at rest and the ability to read their own data.
• Your bucket, no encryption is fully readable: artifacts are written as plain Parquet, protected only by your own bucket access controls. Choose this only if you don’t need DataRecs encryption at rest.

Note

All four postures keep your records encrypted in the database independently — the posture only governs the reconciliation artifacts (Parquet extracts, comparison results, drill-down files) written during a job.

Prerequisites per posture

• The three “Your bucket” postures all require Bring Your Own Storage configured first (a validated GCS bucket).
• Your bucket, your key (BYOK) additionally requires Bring Your Own Key to be configured — the option stays disabled until a verified Cloud KMS key is in place.

Switching posture triggers a background migration

Changing your posture re-encrypts and/or relocates every existing artifact to match the new posture. This runs as a background migration — the same locked, verify-before-delete pipeline as a BYOS storage migration:

• Your data stays readable throughout; the switch only completes once every artifact has moved.
• You can’t change storage or posture settings while the migration is running.
• For large tenants this can take a while — track progress on the storage settings page.

Console

1. Go to Settings → Storage.
2. In Artifact Encryption Posture, pick the posture you want. Disabled options show an inline hint for what to configure first (BYOS or BYOK).
3. Review the confirmation dialog — it states whether you’ll be able to read the artifacts after the switch — then confirm.
4. The migration starts; progress is shown on the same page.

REST API

curl -X POST https://{your-api-domain}/data-storage/artifact-posture \
  -H "Authorization: Bearer {your-api-key}" \
  -H "Content-Type: application/json" \
  -d '{ "posture": "byos-tenant-encrypted" }'

Valid posture values: platform, byos-platform-encrypted, byos-tenant-encrypted, byos-plaintext. The request is rejected (HTTP 400) with BYOS_NOT_CONFIGURED if a “your bucket” posture is chosen before BYOS is set up, or BYOK_NOT_CONFIGURED if byos-tenant-encrypted is chosen before BYOK is configured.

Safety guarantees

• Atomic switchover: Your storage_mode only changes after the copy AND verification both succeed. If anything fails, you stay on your current storage.
• Automatic recovery: If the migration service encounters a transient issue, it automatically retries. If it cannot complete, it aborts cleanly and your data stays where it was.
• No data loss: The source data is never deleted during migration. After a successful forward migration, your data exists in both locations until platform retention policies clean up the platform copy.

Prerequisites

Before starting a BYOS migration:

1. A paid plan — BYOS is available on a paid plan; contact us if it isn’t enabled for your account.
2. A GCS bucket in your GCP project, in a supported region.
3. IAM permissions configured on the bucket for the DataRecs service account.

GCS bucket setup

Create a bucket in your GCP project:

gcloud storage buckets create gs://my-company-datarecs-data \
  --project=my-gcp-project \
  --location=us-central1 \
  --uniform-bucket-level-access

Tip

Choose a region geographically close to your DataRecs cell for best performance. Contact your account team if you’re unsure which region your cell is in.

IAM permissions

Grant the DataRecs tenant service account access to your bucket. The service account follows the pattern:

tenant-{your-tenant-id}@{datarecs-project}.iam.gserviceaccount.com

You can find your exact service account in the Console under Settings → Storage → Service Account, or from the /data-storage API endpoint.

Grant these roles on the bucket:

# Replace with your actual values (find your exact SA in Settings → Storage)
SA="tenant-abc123@<datarecs-project>.iam.gserviceaccount.com"
BUCKET="my-company-datarecs-data"

# Object Admin — read, write, delete objects
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:$SA" \
  --role="roles/storage.objectAdmin"

# Legacy Bucket Owner — required for bucket-level listing
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:$SA" \
  --role="roles/storage.legacyBucketOwner"

Caution

Both roles are required. objectAdmin alone is not sufficient — the migration needs bucket-level list access that only legacyBucketOwner provides.

Step 1 — Configure your storage settings

Tell DataRecs about your GCS bucket so it can validate access.

Console

1. Navigate to Settings → Storage.
2. Click Configure BYOS.
3. Enter your GCP Project ID, Bucket Name, and Bucket Region.
4. Click Validate — DataRecs performs a write/read/delete test on your bucket to confirm permissions.
5. Once validated, click Save Configuration.

REST API

curl -X POST https://{your-api-domain}/data-storage/config \
  -H "Authorization: Bearer {your-api-key}" \
  -H "Content-Type: application/json" \
  -d '{
    "gcp_project_id": "my-gcp-project",
    "bucket_name": "my-company-datarecs-data",
    "bucket_region": "us-central1"
  }'

A successful response returns your current storage configuration (configuring the bucket runs the access check; if it fails the request is rejected):

{
  "storage_mode": "Platform",
  "storage_provider": "GCS",
  "byos_gcp_project_id": "my-gcp-project",
  "byos_bucket_name": "my-company-datarecs-data",
  "byos_bucket_region": "us-central1",
  "byos_migration_status": "not_started",
  "tenant_service_account": "tenant-abc123@datarecs.iam.gserviceaccount.com"
}

Terraform

resource "datarecs_storage_bucket" "byos" {
  gcp_project_id = "my-gcp-project"
  bucket_name    = "my-company-datarecs-data"
  bucket_region  = "us-central1"
}

Step 2 — Start the migration

Once your configuration is validated, trigger the migration.

Console

1. Go to Settings → Storage.
2. Click Start Migration.
3. Review the confirmation dialog (shows estimated data volume and expected duration).
4. Click Confirm.

REST API

curl -X POST https://{your-api-domain}/data-storage/migrate \
  -H "Authorization: Bearer {your-api-key}" \
  -H "Content-Type: application/json" \
  -d '{}'

Response:

{
  "status": "in_progress",
  "message": "Storage migration started. Reconciliation data is being copied to your GCS bucket."
}

Note

Only one migration can run at a time. Attempting a second migration while one is in progress returns HTTP 409.

Step 3 — Monitor progress

Console

The Settings → Storage page shows real-time migration status with a progress indicator.

REST API

Poll the storage endpoint:

curl -H "Authorization: Bearer {your-api-key}" \
  https://{your-api-domain}/data-storage

Status values:

• in_progress — migration is running
• completed — migration finished successfully
• failed — migration encountered an error (see byos_migration_error)

During the migration

• Reads continue normally — your API calls, dashboard, and reports all work.
• Writes are paused — attempts to create or modify reconciliation data return HTTP 503 with a retry-after header. This is temporary and lifts automatically when the migration completes.
• Scheduled jobs are suspended — scheduled reconciliation jobs are paused and resume automatically after migration.

Step 4 — Verify completion

Once the migration reports completed:

• storage_mode is now BYOS
• All reconciliation data artifacts are in your GCS bucket
• Writes are unpaused — new reconciliation results go directly to your bucket
• Scheduled jobs have resumed

You can verify your data is in GCS:

gcloud storage ls gs://my-company-datarecs-data/tenants/{your-tenant-id}/ --recursive

Reverting to platform storage

If you need to switch back to platform-managed storage:

Console

1. Go to Settings → Storage.
2. Click Revert to Platform Storage.
3. Confirm the action.

REST API

curl -X POST https://{your-api-domain}/data-storage/revert \
  -H "Authorization: Bearer {your-api-key}" \
  -H "Content-Type: application/json" \
  -d '{}'

The revert follows the same phases as the forward migration: lock → pause → drain → copy (GCS → Garage) → verify → finalize. Your storage_mode returns to Platform only after all data is verified back in platform storage.

What happens if something goes wrong

Permission revoked during migration

If you revoke the service account’s GCS access while a migration is running:

• The copy step fails with a permission error.
• The migration is automatically aborted.
• Your storage_mode stays unchanged (on whatever it was before).
• Writes are unpaused and scheduled jobs resume.
• The error is recorded in byos_migration_error.

To retry: restore the IAM permissions, then trigger the migration again.

Service interruption during migration

If the DataRecs platform experiences an interruption during your migration:

• The migration has built-in crash recovery.
• Within approximately 2 minutes of the interruption, the system automatically detects the stale migration and cleans up: unpauses writes, resumes scheduled jobs, and releases the lock.
• Your storage_mode stays unchanged.
• You can safely trigger the migration again once the platform is back.

Migration takes too long

Migrations have a maximum runtime (scaled to your data volume, up to 2 hours for very large tenants). If the maximum is exceeded:

• The migration is automatically aborted.
• Cleanup happens as described above.
• Contact support if your data volume exceeds what the standard migration window can handle.

FAQ

Q: Can I use a bucket in a different GCP project than where DataRecs runs?

Yes. The bucket can be in any GCP project, as long as you grant the DataRecs service account the required roles on the bucket.

Q: Is my data deleted from platform storage after migration?

Not immediately. Platform storage retains your data per standard retention policies. This ensures that a revert is always possible without data loss.

Q: What about encryption?

Your data is encrypted by GCS using Google-managed keys (or your own CMEK if configured on the bucket). This is independent of DataRecs’ application-level envelope encryption, which protects data at the record level regardless of where it’s stored. To control how the reconciliation artifacts themselves are encrypted — including making them readable to you with your own KMS key — see Artifact encryption posture above.

Q: Can I restrict which regions my data is stored in?

Yes — you control the bucket’s location settings. DataRecs writes data to whatever bucket you configure, respecting your region choices.

Q: Will my reconciliation jobs be slower after migrating to BYOS?

Performance depends on network proximity between your DataRecs cell and your GCS bucket. If both are in the same or a nearby region, performance is comparable to platform storage. Cross-region configurations may add latency.

Q: How often can I migrate?

As often as needed. Each migration (forward or revert) takes a few minutes. However, writes are paused during migration, so avoid triggering migrations during business-critical reconciliation windows.

Troubleshooting

When configuring a bucket fails, the response carries an error_code describing the access problem. Migration-control and posture-control failures surface as HTTP status codes.

Symptom	Meaning	Resolution
409 on migrate	A migration is already running.	Wait for it to complete, or contact support if it appears stuck.
400 on migrate	BYOS bucket configuration is missing or invalid.	Configure your bucket in Settings → Storage first.
BYOS_NOT_CONFIGURED (400) on set-posture	You chose a “your bucket” encryption posture before configuring BYOS.	Configure Bring Your Own Storage first, then set the posture.
BYOK_NOT_CONFIGURED (400) on set-posture	You chose the byos-tenant-encrypted (your-key) posture before configuring BYOK.	Configure Bring Your Own Key first, then set the posture.
error_code: BUCKET_NOT_FOUND	The named bucket doesn’t exist or isn’t reachable.	Check the bucket name and that it exists in the given project.
error_code: INSUFFICIENT_PERMISSIONS	The service account can’t read/write the bucket.	Grant roles/storage.objectAdmin.
error_code: LIFECYCLE_PERMISSION_MISSING	Missing bucket-level access the migration needs.	Grant roles/storage.legacyBucketOwner in addition to objectAdmin.
error_code: BUCKET_NOT_EMPTY	The target bucket already contains objects.	Use a dedicated, empty bucket for DataRecs.
error_code: SA_IMPERSONATION_FAILED	The DataRecs service account couldn’t be impersonated.	Re-check the IAM bindings on the service account and bucket.
error_code: STORAGE_UNAVAILABLE	GCS was temporarily unreachable.	Transient — retry shortly.
Write returns 503	The write fence is active during migration.	Wait for migration to complete. Reads still work.