Text to Music: Prompting a Generated Track

What the model is generating

A music generation model is not sequencing MIDI and rendering instruments. It generates audio directly, usually as a sequence of learned tokens that a decoder turns into a waveform, which means the arrangement, the performance and the recording all come out already fused. You cannot mute the bass afterwards, because there is no bass track, only a mix that sounds like one.

That single fact explains most of the behaviour that surprises people. Tempo is consistent because the model learned that music has a pulse, not because a clock is running. Instruments enter and leave plausibly because that is what the training data does, not because you arranged them. And you cannot ask for stems from something that was never separate, which is why stem separation is a distinct step run afterwards.

Duration is a real constraint rather than a formality. Models have a window they were trained to fill, and asking for much more than that produces either a hard stop or a drift where the piece loses its own key and pulse.

Prompting for musical facts

The reliable parts of a prompt are the ones a session musician could act on, and the unreliable parts are the ones a client would say.

• Tempo. Give a number: 68 bpm. Words like slow are interpreted across a range of thirty beats.
• Key and mode. D minor costs nothing to specify and strongly shapes the result.
• Named instruments. solo cello, felt piano, upright bass. Named instruments are followed far more closely than genre labels.
• Exclusions. no drums, no vocals. Negative statements are among the strongest levers available, because the default arrangement tends to be fuller than you want.
• One structural event. a slow felt piano entering at the halfway point gives the piece somewhere to go. Asking for four sections usually produces mush.
• Recording character. warm analogue tape, close-miked, dry. This governs the mix as much as the notes.

Genre words are useful as a starting anchor and weak as a description, since any genre label covers arrangements that sound nothing like each other. Use one, then immediately constrain it with instruments and tempo.

Where it fits in a video project

For most video work the deciding question is what the music has to do against picture, and that determines which approach to take.

Underscore that sits below dialogue is the strongest use. Ask for sparse instrumentation, no vocals, and a single build, then generate several takes at the exact duration of your cut. Because generation is cheap, choosing between eight takes beats trying to refine one.

Anything that has to hit picture is harder. Models do not take a timecode, so a sting on a specific frame has to be produced by generating longer than needed and cutting to fit. Plan to trim rather than to prompt.

Songs with lyrics are the case where the separate lyrics field earns its place. Write the words there, keep the style prompt purely musical, and expect to re-roll for diction. Vowels survive better than consonants, and unusual words are where a vocal take tends to fall apart.

The last practical note is loudness. Generated tracks often arrive close to full scale with little headroom, so a track that sounds fine alone will fight dialogue in a mix. Pulling it down and then re-checking under voice is a routine step, not a sign that the generation was bad.