Uploading and Publishing Datasets

To understand the different roles on the DataHive platform, please see the user roles section.

Please follow these steps to upload files and publish them as datasets on DataHive:

No.	Step	How?	Who can do this?
1	Create a Tenant	DataHive has multiple tenants, each with its own set of datasets. Create a tenant here. If you already have a tenant, skip to step 2.	Platform Admins, Chief Data Officers, CDO Team Managers or CDO Team Members
2	Create a Dataset for a Tenant	Create a dataset here.	Tenant Managers
3	Upload or Share Dataset	For tenants not configured for Snowflake Private Listing, upload files to a dataset here. For tenants configured for Snowflake Private Listing, share via Snowflake Private Listing here.	Tenant Admins
4	Update Metadata and Publish the Dataset	Update the metadata and publish the dataset here. Once all files have been uploaded and respective metadata is updated, publishing the dataset will allow DataHive users to see and make requests to the dataset.	Tenant Admins

FAQs​

What file extensions are supported for uploading?​

Currently, only CSV files are supported.

How many files can I upload?​

No limit at this time.

What is the maximum file size for uploads?​

Each CSV file can be up to 2GB in size (a limitation of Microsoft Defender antivirus scanning which supports a maximum file size of 2GB). If your file exceeds this size, please divide it into smaller parts and upload them separately.

How long does it take to publish my dataset?​

A successfully published dataset typically becomes available for subscription within 30 minutes.

How will I know if my dataset has been published to the DataHive portal?​

Tenant Administrators will be notified via email if the publishing process succeeds or fails.

Is it possible to create multiple datasets?​

Yes, there is no limit to the number of datasets that can be created. However, please note that uploaded files are specific to the dataset they are uploaded to and cannot be shared across multiple datasets. If you need the same files in a different dataset, you will need to upload them again.

Can I upload encrypted files?​

Not currently.

Can I limit the visibility of my published dataset to users within my own agency?​

Limiting visibility of published datasets is not currently supported. All published datasets are available for subscription to all whole-of-government users and cannot be restricted to a specific agency or a subgroup of users.

My agency has the same dataset uploaded to Vault. How can we migrate the Vault data requests associated with this dataset?​

Please enter your existing Vault dataset ID on the same page where you have uploaded your file(s). DataHive will port the data requests on a best-effort basis.

Can the same person have both the Tenant Manager and Tenant Admin roles?​

No, the Tenant Manager and Tenant Admin roles must be assigned to separate individuals.

What is the maximum RCST and ISF classification supported?​

RCST: Confidential (Cloud Eligible), ISF: Sensitive (Normal)

Can I publish dataset metadata without the corresponding data?​

No. DataHive does not list datasets that requestors cannot subscribe to. Metadata can only be published with corresponding data.

What is a tenant connection and how do I get one?​

A Tenant Connection is a type of system connection that allows you to upload files programmatically to datasets under a specific Tenant in DataHive. To set one up, you first need a Service Principal Account. Follow the user guide to create one and request access. You only need a Tenant Connection if your system needs to programmatically send data into DataHive.

Can I upload files to a published dataset?​

Yes, you may upload files to both published and unpublished datasets.

How will I know the outcome of my automated file uploads?​

DataHive will send email notifications to all Service Principal Members for each file upload. Additionally, Tenant Admins will receive email notifications when a file upload is successful.

Error: "Failed to retrieve SAS token: 403 Client Error: Forbidden for url:..."​

Please contact the Tenant Admin to assign the dataset to your Service Principal Account.

What methods are available for programmatic uploading of datasets?​

Currently, DataHive only supports programmatic uploads via API.

Can I download my uploaded data?​

No. If the dataset is published, you may make a data request to your dataset to review the data that you uploaded.

Attempting to upload files via API returns me 429 "Too Many Requests" status code​

Whenever a file is uploaded programmatically, an ingestion job will be triggered on DataHive backend. At any 1 time, only 1 ingestion job is allowed per dataset.

Systems are advised to batch their file uploads to avoid hitting the ingestion job limit.

Here are the possible error messages that you may encounter when you try to call the GET /datasets/{dataset_id}/get_upload-link_spn endpoint:

Error message	Remediation action
There is an existing file upload undergoing antivirus scan. Please try again later.	Wait for the file upload ingestion to be completed before uploading again.
There is an existing upload link that has not been used: active_blob_link	Use the provided upload link in the error message to upload your file.
There is an existing file upload ingestion ongoing. Please try again later.	Wait for the file upload ingestion to be completed before uploading again.