--- title: Data delivery standards slug: analysis/data-delivery-standards docTags: createdAt: 2026-01-23T12:02:56.304Z --- ## 1. Delivery Frequency and Reliability To ensure timely and efficient data processing, partners must finalize the following operational parameters with Klavi during the onboarding phase: - **Scope & Historical Data**: Partners are required to coordinate with the **Klavi Business Team** to define the specific data scope and the historical look-back period required for the initial data load. - **Incremental Update Frequency**: Establish the delivery cadence (e.g., daily, weekly, biweekly or monthly) for incremental updates based on your business needs. - **Customized SLAs**: Because data volume and complexity vary significantly between different partners, **Service Level Agreements (SLAs)**—including data delivery cut-off times—are customized. These are determined based on the specific volume of data being processed to ensure realistic and reliable delivery windows. ## 2. Schema Customization\&Data Dictionary Klavi provides flexible data structures tailored to the specific business focus of our partners: - **Customizable Schemas**: Data fields and definitions are not fixed. Partners can customize the schema to align with their internal analytics or reporting requirements. - **Business Consultation**: The final **Data Dictionary** and field mappings will be documented and confirmed in collaboration with Klavi business personnel prior to technical implementation. ## 3. Storage Architecture and Pathing Rules While partners may specify their own **S3 Bucket** and **Base Path** for data reception, Klavi enforces a standardized directory and naming convention for the sub-paths to ensure automated ingestion compatibility. ### 3.1 Directory Structure Data is organized by source type, version, and delivery date using the following partition logic: :::BlockQuote \[Base-Path]/external\_sources\_klavi/\[data\_type]/v\[\]/delivery\_ts=YYYY-MM-DDTHH-MM-SSZ/ ::: - **Example:** :::BlockQuote s3://partner-bucket/custom-prefix/external\_sources\_klavi/credit\_transactions/v1/delivery\_ts=2026-01-23T17-44-58Z/ ::: ### 3.2 Filename Convention Files are named using the data type and a Unix timestamp to ensure uniqueness and auditability: :::BlockQuote \[data\_type]\_\[unix\_timestamp].parquet ::: - **Example:** :::BlockQuote credit\_transactions\_1737557926.parquet ::: ## 4. Technical Specifications We utilize high-performance storage formats to ensure compatibility with modern data warehouses (like Snowflake, BigQuery, or Redshift). - **Format & Compression**: All data is delivered in **Apache Parquet** format using **ZSTD compression**. - **Strict Data Typing: **Columns are defined with appropriate native types (Integer, Boolean, Timestamp). We avoid generic string types to ensure data integrity. - **Raw Data Integrity**: String values (such as Merchant names) are delivered as **raw strings** without normalization or pre-processing to ensure the Partner receives the original source of truth. ## 5. Data Integrity and Idempotency To ensure reliability and prevent ingestion of partial data, we implement the following: - **Immutability**: Delivered data is immutable. Any re-delivery or correction will be issued under a **new timestamped folder**. - **Completion Signal**: A \_SUCCESS marker file is uploaded to the delivery folder **only after** the data upload is 100% complete. Ingestion pipelines should be triggered by the appearance of this file. - **Timezone Standardization**: Time/date fields are the raw values returned by the institution, and Klavi does not perform any conversion. ## 6. Record Management and Deduplication - **Record Deduplication:** The incremental data and historical data distributed to you by Klavi have been removed, and you do not need to consider the removal work. - **Record delay**: Due to the delay of 1-2 days in some institutions' credit card transactions, Klavi will include the updated data in the file next day.