Steps for Configuring a BigQuery Data Pipeline
To configure a BigQuery data pipeline with Intentwise Analytics Cloud, follow these steps:
Sign in to your Google console at https://console.cloud.google.com
In the top-left corner, click Menu > IAM & Admin > Create a Project
Now, enter the project name, choose the organization & location, and click Create. Project Name and Location can be of your choice
Navigate to the BigQuery page. Choose the newly created project and create datasets as mentioned below:
amazon_source_data
intentwise_ecommerce_graph
amazon_marketing_cloud (recommended if you are currently using or intend to use Amazon Marketing Cloud)
criteo_source_data (recommended if you are currently using or intend to use Criteo)
instacart_source_data (recommended if you are currently using or intend to use Instacart)
walmart_source_data (recommended if you are currently using or intend to use Walmart)
You can add the datasets under Explorer and clicking on the three vertical dots icon against the newly created project
Create cloud storage bucket
To create the bucket in Google cloud storage follow these steps:
Choose the newly created project.
Navigate to the Cloud Storage Buckets page.
Click Create
In the Get Started Page Enter globally unique name of your choice for the folder.
Click Continue
In the "Choose where to store your data" section, select the same location as the project, for example: us-west1.
Click "Create" at the bottom of the page. You can skip the other options like "Choose Storage Class" and "Control Access."
Steps for creating service account
Choose the newly created project.
Open the Burger Menu on the side and Go to IAM -> Service Accounts.
In the Service Accounts page, Click on the Create Service Account button on the top. You should now see a form to create a service account. Fill in any Service Account Name, Service Account ID, Service Account Description. Below is an example –
Click "Create and Continue." You should now see an option to assign Service Account permissions. Under that, there will be a dropdown. Search for the roles listed below and add them. Use "Add Another Role" to include each of the following roles:
BigQuery Data Editor
BigQuery Job User
Storage Object User
BigQuery Read Session User
Storage Admin:
When adding Storage Admin Click on Add IAM Condition
Enter any title
In the Condition builder choose below options:
Condition type - Name
Operator - is
Value - projects/_/buckets/bucket-name. Replace only the bucket-name with the bucket created in the previous step. Example: projects/_/buckets/iw_bucket
Once all the Roles are assigned click Done
On the next screen, Click on the newly created Service account email. Choose the KEYS tab. Click on Add Key
Choose Key type JSON and click Create.
Now, login to the Intentwise platform and navigate to My Connections under Intentwise Analytics Cloud. Click on "+ Add Destination", choose Big Query as the destination and click Next.
Enter the Project ID and the Location, Bucket Name and upload the Service account private key in the next window and click on Save.
You can get the Project ID and Location from the Explorer, by clicking on the three vertical dots icon against the dataset and then choosing the option "Open"
On successful completion, you will see this message:
For any further questions on the pipeline setup, please reach out to us on chat or at [email protected].