You can build hum-to-search for a karaoke catalogue, but you should prove it works before committing to it. Split the work into two stages. The first is a lightweight gate that decides whether the incoming audio is a hum or sung melody at all. The second is a matcher that turns the accepted query into a pitch contour or embedding and ranks songs. Give each stage its own labelled test and its own error budget, and make the go/no-go call from a test set sized by scenario (clean hum, off-key hum, background speech, background music, non-hum noise, device variation) rather than by a total clip count.
Letting people search a song by humming is a recognised retrieval idea: researchers describe humming as an effective and natural way to query a musical audio database. What makes it hard in a product is everything around the melody: who hums, into what microphone, over what background noise, and what the feature does with audio that was never a hum.
Is hum-to-search accurate enough to be worth building?
Two published results show the matching half working on their authors’ own data. In a 2023 query-by-humming study (arXiv 2302.04577), adding denoised versions of the queries to the training set raised retrieval accuracy from 67% to 93%. Google describes its Hum to Search system as reaching a high level of accuracy on a database of over half a million songs. Neither result tells you how a model will do on your users, your devices or your catalogue. That gap is exactly what the test set below is for. TechnoLynx works on this kind of AI engineering for media problem.
It also helps to agree early on how the matcher will be scored. The MIREX Query-by-Singing/Humming evaluation scores a top-10 hit rate: one point when the correct song appears in the top ten candidates, zero otherwise. The MIREX Query-by-Singing/Humming results page also scores mean reciprocal rank over the top ten returns. For a karaoke lookup screen that shows a short list of suggestions, a top-k hit rate is a natural acceptance metric for the matcher, kept separate from the gate’s metrics.
How many background scenarios does a hum test set need before you trust a go/no-go call?
A test set is ready for a go/no-go decision when every scenario that represents a distinct failure mode has enough labelled clips to give a stable pass rate on its own. The count of scenarios follows from the failure modes you have found, and the overall clip total says little if one category has only a handful of examples.
TechnoLynx can define the scenario taxonomy, design the opt-in recording campaign that fills the gaps support tickets leave, build the labelling and curation pipeline, and check per-scenario coverage before the feasibility decision.
Where the clips come from
Support tickets are the obvious starting point, because they show what actually went wrong. Check whether ticket audio exists in enough volume and variety, and whether privacy and consent rules allow reusing it directly. Re-creating ticket cases as clean, consented clips solves the privacy problem but loses some of the acoustic and behavioural variety of real users, so recreations should define failure cases, not stand in for the whole population.
A targeted opt-in campaign fills the gaps for conditions that are rare or hard to source, such as heavy background noise or deliberate non-hum input. Volunteers bring a selection bias: people who agree to record themselves humming may well hum more clearly than the average user. Public corpora show the range. HumTrans collected about 56.22 hours of humming from 10 college students, all music majors or proficient instrumentalists, humming along to a score with the melody playing. That is useful for scale and a poor proxy for a karaoke customer humming from memory. MTG-QBH is closer to real use: 118 recordings from 17 subjects, from no musical experience to amateur musicians, sung a cappella without a reference tone on a basic laptop microphone with no post-processing.
Real humming has to be in the data, for training and for testing. The CHAD authors (2023) found that a model trained on open data does well on sung queries, but a pure search-by-humming setup needs real humming data added to training for acceptable performance. Google reports the same pattern: its first model, trained mostly on sung segments, worked well enough for singing but not for humming or whistling. A test set built only from singing will make a hum feature look better than it is.
A scenario taxonomy for a karaoke hum test set
| Scenario | What it tests | Plausible clip sources |
|---|---|---|
| Clean hum | Matcher accuracy under good conditions | Opt-in recordings; MTG-QBH-style laptop captures |
| Off-key or rhythmically loose hum | Matcher tolerance to imperfect queries | Opt-in recordings from non-musicians |
| Sung query with lyrics | Whether the gate accepts singing as well as humming | Opt-in recordings |
| Hum over background speech | Gate and matcher robustness to talkers nearby | Clean hums mixed with speech, e.g. MUSAN’s twelve-language speech |
| Hum over background music | Whether accompaniment pulls the matcher off the hummed melody | Clean hums mixed with music from MUSAN’s genres |
| Non-hum noise only | Gate false accepts | Noise-only clips with speech removed, as in the ICASSP 2022 DNS Challenge noise set, covering noise types such as typing, fans, kitchen and car noise |
| Speech only | Gate false accepts on voiced, pitched input | MUSAN speech |
| Device and microphone variation | Whether results hold across the hardware users actually have | Opt-in recordings on target phones and laptops |
Sizing each scenario
Public sound-classification benchmarks give a reference for test sizes per class. The FSDnoisy18k test set has between 30 and 72 clips per class. The DCASE 2019 FSDKaggle2019 test set has roughly 50 to 150 clips per class. A classic speech/music discriminator (Scheirer and Slaney, ICASSP 1997) was evaluated on 80 fifteen-second samples per class. The arithmetic shows why the lower end is thin: with 30 clips in a scenario, each misclassified clip moves that scenario’s pass rate by 1/30, about 3.3 percentage points. If your go/no-go threshold and your current result are only a few points apart, a scenario that small cannot settle the decision.
Label quality matters as well as count. In FSDnoisy18k, roughly 60% of the labels in the noisy web-sourced portion showed some kind of label noise, and the test set was drawn only from the clean, manually verified portion. Bulk or weakly labelled audio can help training, but every clip in a go/no-go test set should be checked by hand.
Split by recording, never by frame. Scheirer and Slaney held out whole audio samples because frames within one sample are strongly correlated, and splitting a sample across training and test would give a wrong estimate of performance on new data. For hum data, the same reasoning argues for holding out whole speakers, so the test measures people the model has never heard. Finally, treat the taxonomy as provisional: a failure mode nobody has reported yet is not in it, and production misfires should be able to add a new category.
Should hum detection be a separate pre-filter or part of the query-by-humming pipeline?
Make it a separate, lightweight pre-filter that sits in front of the retrieval pipeline. Rejecting noise and non-hum audio is a different job from matching a melody. When the two are separate, each can be tested, tuned and updated on its own, and noise never reaches the matcher.
TechnoLynx can define what counts as a valid hum versus ambient input, choose a small detector suited to the target hardware, implement and optimise it at a clean boundary with an existing query-by-humming pipeline, and measure false accept and reject rates and end-to-end latency against a single-stage baseline. In production, the detector and the matcher can then be updated independently.
Published systems are already described in stages. The Cornell query-by-humming system had three components: a pitch-tracking module, a melody database and a query engine that returned a ranked list. The CHAD authors describe query-by-humming systems as typically having an audio transcription module and a search module, and a 2023 study frames the research as melody extraction followed by candidate melody retrieval. MIREX went as far as encouraging participants to submit separate tracker and matcher modules instead of integrated ones, so that algorithms could share intermediate steps.
The matcher’s results depend on receiving a usable melody. The Cornell authors state that their method’s effectiveness depends directly on how accurately the hummed pitches can be tracked. The 2023 study notes that user queries carry noise and distortion, such as stray microphone clicks, that produce spurious pitch peaks, and that background noise has to be removed to recover the underlying melody. A contour or DTW matcher given a dishwasher recording will still return a ranked list, which gives the user a confident wrong answer.
What the two-stage split costs compared with one end-to-end model
What follows on engineering time and latency for staged versus end-to-end hum search is engineering judgement. A small gating model adds one inference step. It is cheap compared with matching against a catalogue, but it is not free and needs a place in the latency budget. In exchange, the gate can be retrained on new noise or new devices without touching the matcher, and the catalogue or matcher can change without retraining the gate. If gating is folded into the retrieval model, every catalogue or matching change becomes a retraining and revalidation of both behaviours together.
Two risks come with the split. An over-aggressive gate rejects quiet or off-pitch hums, which loses recall the matcher could have recovered. The two stages also need identical audio preprocessing (sample rate, gain normalisation, framing), because a gate tuned on one front end and a matcher fed by another will disagree in ways that are hard to debug. A combined model is worth considering only when the latency budget is extremely tight and a single model is shown on the scenario test set to match the staged pair’s accuracy; otherwise the split carries less engineering risk.
How Google Hum to Search and SoundHound deal with noisy input
Google has published a description of how Hum to Search works. Hum to Search produces an embedding of a melody directly from a spectrogram, with no intermediate representation, and matches hummed audio to the original polyphonic recordings without a hummed or MIDI version of each track. The network is trained so that audio pairs with the same melody sit close together even when their instrumental accompaniment and singing voices differ. To cope with the shortage of real humming, Google generated simulated hummed melodies from existing audio using its SPICE pitch-extraction model.
So Google trains robustness to accompaniment and voice into the matching model itself. The blog post does not describe how, or whether, Google rejects audio that is not a melody at all. For SoundHound, none of our sources documents how noisy or ambient input is handled. Any comparison with either system is therefore approximate. What does carry over is the data lesson: both Google and the CHAD authors needed hum-specific data before humming worked.
Is voice activity detection plus pitch confidence enough to tell a hum from speech or noise?
Not if misfires matter. Voice activity detection (VAD) combined with pitch-tracking confidence makes a good, cheap first pass, but it cannot reliably separate a hummed melody from pitched speech or from background music with vocals. Add a lightweight learned classifier on top for the ambiguous cases. Acceptance is then defined as target false-positive and false-negative rates on held-out labelled clips, not as one acoustic threshold.
TechnoLynx can work with product stakeholders to define the hum, sing, speech and ambient classes and the acceptance metrics, specify the VAD-plus-pitch gate and the classifier behind it, train the classifier for the target runtime (on-device or server) and latency budget, and measure precision, recall and false-positive rate on held-out clips. After launch, misfire monitoring feeds mislabelled cases back into retraining.
The first gap is in VAD itself. The HumAware-VAD model card notes that standard VAD models, Silero-VAD included, often classify humming as speech. A VAD signal therefore shows that someone is vocalising, not that they are humming. Pitch confidence does not close the gap either, because voiced speech and sung background vocals are pitched too, so a high-confidence pitch track is necessary but not sufficient.
Acoustic features that separate a hummed melody from speech and noise
The speech/music literature combines several features in one classifier. Speech has a characteristic energy modulation peak around the 4 Hz syllabic rate. Speech alternates between transitions and relatively stable vowels, while music changes at a more constant rate, which shows up in frame-to-frame spectral difference. Scheirer and Slaney evaluated thirteen such features and found that not all of them added value to a multivariate classifier. Distinguishing humming from singing uses a different cue again: humming tends to contain fewer distinct phones than singing.
Why a small classifier is usually enough
Scheirer and Slaney’s speech/music discriminator reached 5.8% frame-by-frame error, falling to 1.4% when decisions were integrated over 2.4-second segments. They found very little difference between classifier types, which suggested a simple feature space suited to computationally cheap algorithms. That was speech against music, not hum against everything, so treat it as evidence about model size, not as a forecast for your gate. The segment-level result also matters for the design: deciding over a short window rather than per frame removes many isolated errors.
The classifier also needs its own error budget, because even singing-versus-humming discrimination makes errors. An ISMIR 2010 system used a threshold classifier, trained on 80 hand-labelled clips, to separate singing from humming, with the threshold set so that false-positive and false-negative rates were equal. On 1,183 unseen clips it recognised 78.61% correctly, with 10.99% of humming clips misclassified as singing and 24.51% of singing clips misclassified as humming. The authors judged the second error less harmful because melody recognition was already accurate. That asymmetry is the right way to think about any gate: decide which error costs the user more and set the threshold for that, rather than defaulting to equal error rates.
Choosing the gate
| Situation | Gate design | What acceptance looks like |
|---|---|---|
| Explicit tap-to-hum, tight compute, occasional misfire tolerable | VAD plus pitch-salience or confidence threshold only | False-accept rate on noise and speech clips within the agreed budget |
| Misfires are costly, or input includes speech or vocal background music | VAD plus pitch gate, then a lightweight learned classifier | Separate false-accept and false-reject targets per scenario on held-out clips |
| Lyrics are used as an extra matching signal | Add a singing/humming discriminator after the gate, as in the ISMIR 2010 design | Error budget for hum-versus-sing confusion, weighted by which error hurts more |
The trade-off is plain: a classifier adds inference cost and a dependency on labelled data that a pure signal-processing gate avoids. Thresholds and classifier performance also depend on the corpus, so they need revalidating on the devices, microphones and languages of the real user base.
What false-positive rate on non-hum audio is acceptable?
There is no reason to expect a universal acceptable false-positive rate. The acceptable false-positive rate is a product decision, set by how much misfiring users will tolerate, not by model metrics alone. Here the interface sets the tolerance. With an explicit tap-to-hum button, a false accept costs one bad result list after the user chose to search. With always-on listening, every false accept is the feature triggering when nobody asked it to, and the tolerance is far lower. Write the threshold down as a pair of rates (false accepts on non-hum scenarios, false rejects on valid hums) measured on the held-out scenario set, agreed with product owners before the go/no-go test runs, so the result cannot be reinterpreted after the fact.
The catalogue side decides what the matcher can find
For karaoke specifically, the reference side needs as much thought as the query side. Karaoke backing tracks leave out the lead vocal, which is exactly the melody a user hums. The CHAD authors state plainly that their approach cannot search instrumental tracks. Classic query-by-humming evaluation matched queries against monophonic ground-truth MIDI files. Before scoring any hum gate, check what melody reference each catalogue entry actually has: a guide vocal, a MIDI melody line, the original recording, or nothing. Songs with no usable melody reference cannot be found by humming, however good the gate is, and that share of the catalogue belongs in the go/no-go call alongside the error rates.
FAQ
Which machine learning approaches are used to tell speech from music?
The Scheirer and Slaney discriminator combined several hand-crafted features (syllabic-rate modulation energy, spectral flux and their variances) in simple classifiers such as Gaussian models, nearest-neighbour or spatial partitioning. Corpora such as MUSAN are suitable for training VAD and music/speech discrimination models.
