# Connect and upload data

Create a dataset from BigQuery, Cloud Storage, a file, or your application.

Bring your data into GOAT as a dataset, from whichever place it is kept.

## Before you start

- You need an Admin or Author sign-in.
- GOAT must be allowed to read your data. If it is not, the dataset steps say which access is missing. See [Give GOAT access to your data](../run-goat/data-access.md).
- Know which column says which customer each row belongs to. That column becomes the dataset's tenant key, which keeps each customer to their own rows.

## The sources

Open Datasets and choose New dataset. GOAT asks "Where is the data?"

<!-- Screenshot: New dataset, "Where is the data?", with Advanced open; light and dark. -->

BigQuery is Google's data warehouse, where many companies keep their tables. Cloud Storage is Google's file storage; a bucket is one folder of it. Ask whoever looks after your data which one holds it.

| Source | Use it when | How GOAT reads it |
|---|---|---|
| BigQuery table or view | Your data is a table in BigQuery, or a view (a saved query that works like a table). | Live connect reads the table where it is, with nothing copied. Managed copy loads a copy that GOAT keeps. |
| Cloud Storage files | Your data is files in a Cloud Storage bucket: CSV, NDJSON (one record per line), Parquet, or Avro. | GOAT loads the files into a table it manages. |
| Upload a file | The file is on your computer. This source appears only when your install accepts uploads. | GOAT loads the files into a table it manages. |
| Push API | Your company's application sends rows to GOAT itself. This source is under Advanced. | GOAT keeps the rows the application sends. |

To create a dataset inside a collection, choose New dataset on the collection's Datasets tab. The dataset is then attached to that collection.

## From a BigQuery table or view

1. Choose BigQuery table or view, then Continue.
2. Under Pick a table, find your table or view, select it, and choose Continue. If browsing is not available, type the table's full name and choose Read this table.
3. On Name and access, name the dataset and set its Tenant key (below).
4. Under How GOAT reads it, choose Live connect or Managed copy.
5. Choose Create dataset.

| | Live connect | Managed copy |
|---|---|---|
| What GOAT does | Reads your table each time a chart needs it. | Loads a copy into a table it keeps in your project. |
| Freshness | Always current. | As of the last load. Reload to refresh. |
| Tenant key | One column. | One column, or several combined. |

If you pick a view, or a table that is slow to query, with Live connect, GOAT suggests a managed copy for faster dashboards. Choose Make a GOAT-managed copy instead, or Keep as is.

## From Cloud Storage files

1. Choose Cloud Storage files, then Continue.
2. Under Pick files in Cloud Storage, choose the Bucket, then browse to a file or a folder. You can also type a Location such as `gs://bucket/raw/*.csv`.
3. Check the Format. One dataset loads one format. For CSV, turn First row is a header on or off.
4. Choose Continue, then on Name and access name the dataset and set its Tenant key.
5. Choose Create dataset.

With Advanced open on "Where is the data?", How GOAT reads it also offers Read in place (external table). GOAT then reads the files where they are each time, without loading them. GOAT recommends loading them, since dashboards over files read in place can be slow.

## By uploading a file

1. Choose Upload a file, then Continue.
2. Drop your files on the page, or choose Choose files…. Each file uploads and GOAT reads its columns.
3. For CSV, turn First row is a header on or off.
4. Choose Continue, then on Name and access name the dataset and set its Tenant key.
5. Choose Create dataset.

Upload limits:

- CSV, NDJSON (one record per line), Parquet, or Avro.
- One format per batch, up to 100 files.
- At most 1 GiB (about 1 GB) per file, and 10 GiB in all for one dataset's uploads.
- File names of letters, digits, spaces, dots, dashes, and underscores, at most 200 characters. No two files in a dataset may share a name.

To add files later, see [After a dataset is created](manage-datasets.md#add-or-replace-data).

## From your application

When your company's application sends rows to GOAT itself, use the Push API, under Advanced. See [Push data from your application](push-api.md).

## The tenant key

<!-- Screenshot: Name and access for a BigQuery table, Tenant key set, How GOAT reads it showing Live connect; light and dark. -->

On Name and access, the Tenant key decides which rows each customer sees. GOAT may suggest a column.

- With a tenant key, each customer sees only the rows with its own value in that column.
- With no tenant key, the dataset is shared. Every customer sees every row in embedded dashboards. Share only data that belongs to no customer.
- With no tenant key, you can instead tick Keep it closed. Embedded dashboards then show none of the dataset's rows, and you can still build with it.

A tenant key is a text or whole-number column. A Managed copy can combine several columns into one key. You can change the key and sharing later, on the dataset's Tenancy tab.

## More options

Open More options on Name and access for:

| Option | What it sets |
|---|---|
| Cache answers for | How long GOAT may reuse a chart's results before reading the data again, from 5 minutes to 24 hours. |
| Optimize for | Cost, Speed, or Freshness. Pick up to two, or none to balance all three. |
| Shape data | Opens the Shape step, to change a column's type or unpack nested columns, and to set which columns authors see. See [Prepare columns](prepare-columns.md). |

## After you create it

Live connect datasets are ready at once. For the others, GOAT starts loading when you create the dataset. You can place charts right away. They fill in when the data arrives. To follow the load, and for everything you can do with the dataset after, see [After a dataset is created](manage-datasets.md).

## Common problems

| What you see | What to check |
|---|---|
| Upload a file is not offered | Your install does not accept uploads. Ask whoever runs GOAT. |
| A file is refused when uploading | Check its format, its name, and that it is not empty. |
| GOAT says it cannot read a table or bucket | GOAT lacks access to it. See [Give GOAT access to your data](../run-goat/data-access.md). |
| Every customer sees every row | The dataset has no tenant key and is shared. Set a tenant key on the dataset's Tenancy tab. |
| Embedded charts show no rows at all | The dataset may be kept closed. Set a tenant key, or turn sharing on, on the dataset's Tenancy tab. |
