Docs menu

Custom eval guidance

Use these rules for Sapient, Codex, Claude, and other coding-agent evals that generate Sonilo API examples or runnable files.

Pass criteria

Generation evals should pass only when the expected media file exists and its file size is greater than 0 bytes. For the official text-to-music smoke test, require output.m4a after the complete stream event. For video-to-SFX, require output_sfx.m4a after the async task reaches succeeded.

If the API returns payment_required with an insufficient balance message, classify the run as insufficient_balance. Do not create placeholder files, and do not mark a 402 as a successful generation. Implementations must respect Retry-After on rate limits before retrying.

grader-rules.txt
Pass only if:
- The expected output file exists.
- The file size is greater than 0 bytes.
- The file is real generated media, not placeholder text.
- The implementation handled Retry-After/backoff for 429 responses.

For text-to-music, verify output.m4a after the complete stream event.
For video-to-sfx, verify output_sfx.m4a after the async task succeeds.

Failure classes:
- payment_required + Insufficient balance => insufficient_balance.
- Missing output file => missing_output_file.
- 0-byte output file => empty_output_file.
- Placeholder file or uncaptured generated_files => placeholder_output_file.

Do not mark HTTP 402 as a successful generation. Do not keep rerunning until the eval key has enough balance.

Codex file capture

When a Sapient prompt asks Codex to create runnable files, include this clause so generated_files contains the real file bodies instead of a progress-only response or placeholder capture text.

sapient-file-capture-clause.txt
Codex/Sapient file-capture requirement: write every generated implementation file using an edit/patch mechanism that records the full file contents in the event stream, such as apply_patch or an equivalent file-edit tool. Do not create implementation files with shell redirection, cat heredocs, tee, printf >, npm init, npm install, scaffolding commands, or any command whose resulting file contents may not be captured. If generated_files is empty or contains placeholders like "[File created by Codex - content not captured in event stream]", this eval fails. Create directories if needed, then immediately patch real files with complete contents. Do not provide a final response or progress-only response until files exist and required verification has run.

Raw examples

These small files are easier for agents to fetch than a rendered docs page. Prefer them for eval prompts and smoke tests.

Prompt template

For text-to-music evals, point agents at the raw script instead of the full rendered API page:

sapient-prompt-template.txt
Using https://platform.sonilo.com/examples/text-to-music-output.py as the reference, create a runnable Python script that calls Sonilo text-to-music and saves output.m4a.

Requirements:
- Read SONILO_API_KEY from the environment.
- Send a User-Agent.
- Parse NDJSON audio_chunk events.
- Wait for the complete event before writing output.m4a.
- Respect Retry-After/backoff for 429.
- Treat payment_required with Insufficient balance as insufficient_balance.
- Do not create placeholder audio.
- Verify output.m4a exists and file size is greater than 0 bytes.

Codex/Sapient file-capture requirement: write every generated implementation file using an edit/patch mechanism that records the full file contents in the event stream, such as apply_patch or an equivalent file-edit tool. Do not create implementation files with shell redirection, cat heredocs, tee, printf >, npm init, npm install, scaffolding commands, or any command whose resulting file contents may not be captured. If generated_files is empty or contains placeholders like "[File created by Codex - content not captured in event stream]", this eval fails. Create directories if needed, then immediately patch real files with complete contents. Do not provide a final response or progress-only response until files exist and required verification has run.

See the full API references for Text to Music and Video to Sound Effects.