Schlagwort: gcp

Veröffentlicht am

Google BigQuery: Export all tables to Cloud Storage

Small shell script to export all tables of a BigQuery dataset to cloud storage.

#!/bin/bash

BGPROJECT='myproject01'
BGDATASET='mydataset01'
GSBUCKET='mybucked01'
GSFOLDER='myfolder01'

# remove VIEW and EXTERNAL

BGTABLES=$(bq ls ${BGDATASET} | grep 'TABLE' | awk '{print $1}')


for i in ${BGTABLES}
do
  echo ${i}
  bq extract --destination_format AVRO --compression SNAPPY ${BGDATASET}.${i} gs://${GSBUCKET}/${GSFOLDER}/${i}*.avro
done