ReleaseBytes
gcp Google Cloud Blog ·

BigQuery introduces `%%bqsql` magic for seamless SQL and Python data processing

blogdatagcpgadata-scientistmediagcp-bigquery
feature

Google Cloud introduces the `%%bqsql` IPython cell magic, allowing data scientists and engineers to seamlessly chain data processing workflows across SQL and Python in Jupyter notebooks. This capability eliminates the friction of manually moving data between BigQuery SQL results and Python's in-memory data structures. Expanding on functionality from Colab Enterprise, it enables direct querying of local pandas DataFrames using SQL and saving results back to BigFrames DataFrames. This integration enhances hybrid processing pipelines for scalable data workloads.

  • →Introduce `%%bqsql` IPython magic for SQL-Python integration
  • →Enable seamless data flow between pandas DataFrames and BigQuery SQL
Features (1) ›
  • Introduce `%%bqsql` IPython magic for SQL-Python integration

    The `%%bqsql` IPython cell magic is now available, enabling data scientists and engineers to effortlessly chain data processing workloads between SQL and Python code cells. This functionality expands the seamless integration capabilities previously offered in Colab Enterprise to the broader open-source ecosystem, including Jupyter notebooks.

Enhancements (1) ›
  • Enable seamless data flow between pandas DataFrames and BigQuery SQL

    This new magic eliminates the historical friction of manually moving data between SQL query results and Python's in-memory structures. It allows the BigQuery query engine to directly reference local pandas DataFrames as temporary tables and saves SQL query results as BigFrames DataFrames for immediate use in subsequent Python cells, facilitating multi-step, hybrid processing pipelines.

Read the original announcement →

https://cloud.google.com/blog/products/data-analytics/bridge-sql-and-python-with-bigquery/

Related releases