Custom eval guidance
Use these rules for Sapient, Codex, Claude, and other coding-agent evals that generate Sonilo API examples or runnable files.
Pass criteria
Generation evals should pass only when the expected media file exists and its file size is greater than 0 bytes. For the official text-to-music smoke test, require output.m4a after the complete stream event. For video-to-SFX, require output_sfx.m4a after the async task reaches succeeded.
If the API returns payment_required with an insufficient balance message, classify the run as insufficient_balance. Do not create placeholder files, and do not mark a 402 as a successful generation. Implementations must respect Retry-After on rate limits before retrying.
Pass only if:
- The expected output file exists.
- The file size is greater than 0 bytes.
- The file is real generated media, not placeholder text.
- The implementation handled Retry-After/backoff for 429 responses.
For text-to-music, verify output.m4a after the complete stream event.
For video-to-sfx, verify output_sfx.m4a after the async task succeeds.
Failure classes:
- payment_required + Insufficient balance => insufficient_balance.
- Missing output file => missing_output_file.
- 0-byte output file => empty_output_file.
- Placeholder file or uncaptured generated_files => placeholder_output_file.
Do not mark HTTP 402 as a successful generation. Do not keep rerunning until the eval key has enough balance.Codex file capture
When a Sapient prompt asks Codex to create runnable files, include this clause so generated_files contains the real file bodies instead of a progress-only response or placeholder capture text.
Codex/Sapient file-capture requirement: write every generated implementation file using an edit/patch mechanism that records the full file contents in the event stream, such as apply_patch or an equivalent file-edit tool. Do not create implementation files with shell redirection, cat heredocs, tee, printf >, npm init, npm install, scaffolding commands, or any command whose resulting file contents may not be captured. If generated_files is empty or contains placeholders like "[File created by Codex - content not captured in event stream]", this eval fails. Create directories if needed, then immediately patch real files with complete contents. Do not provide a final response or progress-only response until files exist and required verification has run.Raw examples
These small files are easier for agents to fetch than a rendered docs page. Prefer them for eval prompts and smoke tests.
- /examples/text-to-music-output.py saves
output.m4afrom the text-to-music NDJSON stream. - /examples/video-to-sfx-minimal.py submits
POST /v1/video-to-sfx, pollsGET /v1/tasks, and savesoutput_sfx.m4a. - /examples/account-usage-dashboard.py calls
GET /v1/account/servicesandGET /v1/account/usage. - /examples/sapient-grader-output-file.py implements the output-file grader rule.
Prompt template
For text-to-music evals, point agents at the raw script instead of the full rendered API page:
Using https://platform.sonilo.com/examples/text-to-music-output.py as the reference, create a runnable Python script that calls Sonilo text-to-music and saves output.m4a.
Requirements:
- Read SONILO_API_KEY from the environment.
- Send a User-Agent.
- Parse NDJSON audio_chunk events.
- Wait for the complete event before writing output.m4a.
- Respect Retry-After/backoff for 429.
- Treat payment_required with Insufficient balance as insufficient_balance.
- Do not create placeholder audio.
- Verify output.m4a exists and file size is greater than 0 bytes.
Codex/Sapient file-capture requirement: write every generated implementation file using an edit/patch mechanism that records the full file contents in the event stream, such as apply_patch or an equivalent file-edit tool. Do not create implementation files with shell redirection, cat heredocs, tee, printf >, npm init, npm install, scaffolding commands, or any command whose resulting file contents may not be captured. If generated_files is empty or contains placeholders like "[File created by Codex - content not captured in event stream]", this eval fails. Create directories if needed, then immediately patch real files with complete contents. Do not provide a final response or progress-only response until files exist and required verification has run.See the full API references for Text to Music and Video to Sound Effects.