Skip to content

Comments

Add bigquery pipeline audit prompt#774

Open
ramyashreeshetty wants to merge 2 commits intogithub:stagedfrom
ramyashreeshetty:add-bigquery-pipeline-audit-prompt
Open

Add bigquery pipeline audit prompt#774
ramyashreeshetty wants to merge 2 commits intogithub:stagedfrom
ramyashreeshetty:add-bigquery-pipeline-audit-prompt

Conversation

@ramyashreeshetty
Copy link

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • My contribution adds a new instruction, prompt, agent, or skill file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, or skill with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.

Description

This prompt guides Copilot through a structured 6-section review (Cost Exposure, Dry Run Modes, Backfill/Loop Design, Query Safety, Safe Writes, and Observability) and produces a PASS/FAIL report with exact patch locations ordered by risk. Useful for data engineers who want to catch runaway BigQuery costs, prevent duplicate writes, and ensure pipeline failures are visible before shipping to production.

A standard code review wasn't able to catch these issues, which actually caused me problems in the past. This prompt helped me identify critical audit findings before deploying anything to production. It generates a structured report, what you act on is entirely up to you.

Usage Example:
Ran the bigquery-pipeline-audit prompt against run_backtest_simulation_v2.py, a Python script that runs BigQuery-backed backtest simulations. The audit identified 3 critical cost risks: a date-by-date loop generating up to 240 BQ jobs, missing maximum_bytes_billed limits exposing 6 TB of potential scans, and no idempotency on writes (re-runs create duplicate data). Estimated worst-case cost: ~$30 per backtest run with risk of unlimited growth. The prompt returned a prioritized patch list with exact file locations and function names, failing 5 of 6 audit sections.

image

Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • Update to existing instruction, prompt, agent, plugin, or skill.
  • Other (please specify):

Additional Notes

  • The prompt is designed to be read-only, it analyzes the codebase and returns a structured audit report rather than auto-applying fixes.
  • Sections are modular: reviewers or users can adopt individual checks (e.g., just the cost exposure section) independently.

By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants