feat: add evaluation framework and CI/CD integration for csv-processor sample #1179
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
test:uipath-langchain
…r sample Add comprehensive evaluation framework for testing file input scenarios in the csv-processor sample. ## Changes ### Sample Updates (samples/csv-processor/) - Add Output model to main.py for proper evaluation data access - Create test data (sales_data.csv, large_dataset.csv, minimal.csv) - Add custom evaluators: - CSVShapeEvaluator: validates CSV dimensions (rows × columns) - CSVColumnsEvaluator: verifies expected column names - AttachmentCreatedEvaluator: checks output attachment creation - Add evaluation set (file-input-tests-local.json) with 3 test cases - Update uipath.json with Output model schema ### CI/CD Integration (testcases/csv-processor-evals/) - Add testcase directory structure following existing patterns - Create run.sh script for automated evaluation runs - Add assert.py for validation of evaluation results - Configure pyproject.toml and uipath.json for testcase execution ### Core Fixes - Fix evaluator path resolution bug in _evaluator_factory.py - Support co-located .py and .json files in evaluators directory - Check direct path before falling back to /custom/ subdirectory - Add pandas dev dependency for CSV processing ## Evaluation Results All 3 test cases pass with perfect scores (1.0/1.0): - Test Sales Data CSV Processing - Test Large Dataset CSV Processing - Test Minimal CSV Processing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixed three critical issues causing evaluation tests to fail: 1. Added pandas dependency to testcases/csv-processor-evals/pyproject.toml - Agent requires pandas but it was missing from eval dependencies 2. Fixed file paths in file-input-tests-local.json - Updated paths to be relative to testcases directory - Changed from "test-data/..." to "../../samples/csv-processor/test-data/..." 3. Optimized UiPath initialization in main.py - Moved UiPath() initialization to platform mode only - Prevents authentication errors when testing with local files All tests now pass with scores of 1.0 for both evaluators. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Chibionos
force-pushed
the
feat/csv-processor-evaluations
branch
from
adf54bc to
dc2658c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive evaluation framework for testing file input scenarios in the csv-processor sample.
Changes
Sample Updates (
samples/csv-processor/)Outputmodel to main.py for proper evaluation data accessfile-input-tests-local.json) with 3 test casesuipath.jsonwith Output model schemaCI/CD Integration (
testcases/csv-processor-evals/)run.shscript for automated evaluation runsassert.pyfor validation of evaluation resultspyproject.tomlanduipath.jsonfor testcase executionCore Fixes
_evaluator_factory.py.pyand.jsonfiles in evaluators directory/custom/subdirectoryEvaluation Results
All 3 test cases pass with perfect scores (1.0/1.0):
Test plan
uv run uipath eval main evaluations/eval-sets/file-input-tests-local.json./testcases/csv-processor-evals/run.sh🤖 Generated with Claude Code