Snowflake Secure Data Sharing
DataHive utilizes Snowflake for the warehousing of its data assets.
Snowflake's Secure Data Sharing feature can facilitate a one-way direct connection between users' Snowflake instances and DataHive's Snowflake instance, with the connection being brokered and managed by Snowflake.
Pre-requisites to receive data from DataHive
- Snowflake instance with a Business Critical tier in the Azure Southeast Asia (Singapore) region.
- A fulfilled Data Request (DR) requesting for data from Trusted Center for Sensors Data (TCS).
- As a DR owner, email to support@datahive.gov.sg with your 7 character Snowflake instance ID (e.g. AB12345) to request for the Snowflake-to-Snowflake secure data share.
How does Secure Data Sharing work?
When tables are written to Snowflake, Snowflake manages the data securely using underlying storage accounts in the respective cloud providers (Azure, AWS, Google Cloud).
Since the data is managed by Snowflake, no actual data is copied or transferred between accounts in a Secure Data Share. Snowflake simply brokers a logical access point to the underlying table/view being shared.
There are a few benefits to this approach of data sharing:
- No data staleness. Users will read from the same underlying source when reading from the Secure Data Share. Changes to the table would be reflected near-instanteously on the users' end.
- No storage costs. Since there is no data replication, there is no storage costs for users. The only charges are for the compute resources used to query the shared data.
- Less overhead. Secure Data Sharing is brokered and occurs on the Snowflake control plane. There is no need to setup data extraction/ingestion scripts, whitelist firewalls and/or set up file transfer protocols etc.
Data Shared is read-only. The shared data cannot be modified by the user.
Direct Share vs Private Listing
Snowflake offers two mechanisms for Secure Data Sharing. They differ mainly in whether they can share data across cloud regions and platforms, and whether data is replicated.
Direct Share is the mechanism DataHive uses today. Snowflake brokers a logical access point to the underlying table/view and no data is copied or replicated. Because of this, a direct share can only be consumed by an account in the same Snowflake region and on the same cloud platform as the provider. This is why receiving data from DataHive requires a Snowflake account in the Azure Southeast Asia (Singapore) region (see the pre-requisites above).
Private Listing can share data with consumer accounts in different regions and on different cloud platforms. To achieve this, Snowflake uses Cross-Cloud Auto-fulfillment to automatically replicate the shared data to the consumer's region/cloud. Snowflake manages the replication, but because a copy of the data is provisioned in each target region, it incurs additional replication and storage costs (borne by the provider).
| Aspect | Direct Share | Private Listing |
|---|---|---|
| Data replication | None — logical access to the source only | Snowflake replicates the data to the consumer's region/cloud via Cross-Cloud Auto-fulfillment |
| Cross-region / cross-cloud | Not supported — provider and consumer must be in the same region and cloud platform | Supported |
| Storage footprint & cost | No additional copy; no storage cost for the shared data | A replicated copy is provisioned in each target region, incurring additional storage and replication costs |
| Data freshness | Reads the live source; changes reflected near-instantaneously | Replicated data is refreshed periodically, so there may be a slight lag behind the source |
DataHive's Snowflake instance is hosted in the Azure Southeast Asia (Singapore) region, and DataHive shares data only via a direct share without replicating data across regions or clouds. Data providers are responsible for any cross-cloud replication. The supported sharing scenarios are:
| # | From | To | Supported | Reason |
|---|---|---|---|---|
| 1 | DataHive | Azure Southeast Asia (Singapore) instance | ✅ | Same region and cloud — direct share, no replication needed |
| 2 | Azure Southeast Asia (Singapore) instance | DataHive | ✅ | Same region and cloud — direct share, no replication needed |
| 3 | AWS Asia Pacific (Singapore) instance | DataHive | ✅ | Different cloud platform (even in the same geography) — the AWS-side provider replicates its data to DataHive's Azure Southeast Asia (Singapore) Snowflake instance via Cross-Cloud Auto-fulfillment |
| 4 | DataHive | AWS Asia Pacific (Singapore) instance | ❌ | Different cloud platform — would require DataHive to replicate data across clouds |
Interested in sharing your dataset with DataHive via Secure Data Sharing?
If you have datasets that you believe would benefit whole-of-government users and wish to onboard them to DataHive via Snowflake, please contact us at admin@datahive.gov.sg.