I would like to use the experimental option that allows me to update a BigQuery schema when performing a load job.
I’m using Dataflow and the built-in BigQueryIO.write from the SDK.
I saw that with a JobConfigurationLoad.setSchemaUpdateOptions(ALLOW_FIELD_ADDITION) from the BigQuery API it’s possible, but I can’t find the equivalent with the BigQueryIO.
Does it exist somewhere or can I override some part in the BigQueryIO to do that ?
Thank you very much,
AFAIK, that experimental option is not yet exposed via the Dataflow/Beam APIs in BigQueryIO, and it would not be a trivial task to override something in in that class – I wouldn’t recommended going down that route.
One workaround I can think of would be to redirect your sink to GCS instead of BigQuery, and then perform a normal BigQuery load job(s) at the end of your pipeline. That way you can use the option.