To connect to Google BigQuery, you must install and configure client files to implement communication between ArcGIS and BigQuery and create database connection files from ArcGIS Pro. To publish web layers, the database connection files must be registered with the ArcGIS Enterprise portal's hosting server site. For best performance, deploy ArcGIS clients in Google Cloud.
Typically, IT professionals install the ArcGIS software and the drivers required to connect to the data warehouse. They also configure network, firewall, and other communication channels necessary to allow communication between ArcGIS and the cloud data warehouse.
Next, the administrator of the cloud data warehouse creates accounts and grants the accounts privileges to access specific tables in the cloud data warehouse. The administrator can create a connection from ArcGIS Pro to the cloud data warehouse and share the connection file with publishers, or the administrator can provide publishers with the credentials necessary to allow them to create their own connection files.
To publish ArcGIS Server web services that reference data in the cloud data warehouse, publishers must register the database connection file with the ArcGIS GIS Server site that acts as the ArcGIS Enterprise portal's hosting server.
Install and configure drivers
To connect from ArcGIS Pro to BigQuery, you must install the latest Magnitude Simba Open Database Connectivity (ODBC) driver on the ArcGIS Pro machines. To publish a map image layer or feature layer that contains data from BigQuery, you must install drivers and configure the machines in the ArcGIS Enterprise portal's hosting server site to connect to BigQuery.
See Google BigQuery documentation for more information and links to download and install the 64-bit ODBC driver.
Microsoft Windows machines
Install the Magnitude Simba ODBC driver on each ArcGIS Pro machine and each machine in a hosting server that uses a Windows operating system.
Confirm that the path to the Magnitude Simba ODBC driver is present in the Windows PATH system environment variable. If it is not, add it.
Linux machines
If the portal's hosting server runs on Linux machines, you must install the Magnitude Simba ODBC drivers on each machine in the GIS Server site.
When you edit the Magnitude Simba /etc/odbcinst.ini file to reference the location of the Simba ODBC driver, you must change the section heading to be Simba ODBC Driver for Google BigQuery. The text must match that exactly. For example, if the heading is Simba ODBC Driver for Google BigQuery 64bit or anything else, connections from ArcGIS will fail.
After you configure the /etc/odbcinst.ini file, configure the ArcGIS Server init_user_param.sh file to access the drivers.
Alter the init_user_param.sh script
Alter the init_user_param.sh script installed with ArcGIS Server to reference the drivers. You can access this script by browsing to the <ArcGIS Server installation directory>/arcgis/server/usr directory.
- Ensure that the ArcGIS Server installation owner on each machine has at least read and execute permissions on the data warehouse client libraries.
- Open the init_user_param.sh script in a text editor.
- Remove the comment marks (#) from the lines beginning with export and replace the driver locations with the paths on your machine.
# ----------------------------------------------------------------------- # Modify this section to configure an ODBC connection to Google BigQuery Data # Warehouse. # # To make a connection, copy GoogleBigQueryODBC.did to the lib folder under the # extracted directory for Google BigQuery ODBC driver. Edit # simba.googlebigqueryodbc.ini and set DriverManagerEncoding=UTF-16. Configure and # store the odbcinst.ini in the /etc folder. # # ----------------------------------------------------------------------- export LIB_ODBC_DRIVER_MANAGER=<Location_to_ODBC_driver_manager_libraries>/libodbc.so.2 export SIMBAGOOGLEBIGQUERYODBCINI=<Location_to_simba.googlebigqueryodbc.ini>/simba.googlebigqueryodbc.ini - Save and close the script.
- For the changes you made in the init_user_param.sh file to take effect, you must restart ArcGIS Server.
To do this, run the stopserver.sh script, then run the startserver.sh script on each machine in the ArcGIS Server site.
./stopserver.sh
./startserver.sh
Create database connection files
In ArcGIS Pro, use the New Database Connection dialog box to create a connection to BigQuery.
- Open the Catalog pane in ArcGIS Pro.
- Right-click Databases and click New Database Connection.
- Choose BigQuery from the Database Platform drop-down list.
- From the Authentication Type drop-down list, choose the type of connection to make:
- User Authentication—Uses a refresh token for authentication.
- Service Authentication—Uses a service account key file for authentication.
See Google BigQuery documentation about authentication to help you decide which to use.
- If you choose User Authentication, provide a refresh token.
- If you choose Service Authentication, browse to the location of the key file that contains the credentials you need to connect to BigQuery, select the file, and click Open.
- Type the project ID and name of the default dataset associated with the user you want to connect as.
Note:
Do not connect using any of the BigQuery public data projects as the default project. You cannot enable feature binning and cannot create a materialized view directly on any tables in the datasets of the public data projects.
If you intend to create a materialized view using this connection file, ensure the connection is assigned a default dataset in which database objects can be created.
- If you require parameters to connect in addition to those required in the previous steps, provide the additional parameters in the Advanced Options field.
Separate the options with semicolons (;). Read the Google BigQuery documentation for information on advanced connection parameters.
For example, to configure the ODBC connector to retry failed API calls for 200 seconds and to use a certificate from a certifying authority (CA) stored on the shared drive \\certificates, type Timeout=200;TrustedCerts=\\certificates in the Advanced Options field.
- Click OK to create the connection file.
You can rename the file by typing a new name in the Catalog pane and pressing Enter.
Ensure the account you used for the connection has the correct privileges to access content in the data warehouse.
Register with the hosting server
To allow the ArcGIS Server site to access the data, use the database connection file you created to add a registered data store in ArcGIS Pro or add a data store item in the portal.
Next steps
After you configure the ArcGIS clients to connect and create a database connection file, you and other organization members can do the following:
- If publishers intend to create snapshots of data to be stored in the relational data store, ensure the ArcGIS Data Store administrator is aware of this and has configured sufficient memory and disk space on the relational data store machines and backup machines to accommodate it.
- Analysts and publishers create a query layer in a map in ArcGIS Pro.