Switchyard

The USER layer is a HackerNews thread about NVIDIA NeMo Switchyard (github.com/NVIDIA-NeMo/Switchyard), an experimental open-source framework for routing/switching between LLMs mid-session. Key user findings: (1) The README states 'Experimental software. Not for production use.' while the press release appears to encourage deployment - a contradiction users flagged ('Unclear if they actually want you to deploy it or not, press release says yes, README says no'). (2) A user reviewing the routing strategies found they 'tend to rely on extra llm calls to determine which model to pick,' which makes sense for batch classification with many fine-tuned models but adds cost for conversation-oriented use. (3) Several users worked through the cache economics of mid-session switching (Cursor example: switching GPT to DeepSeek at a 200k context): each model keeps its own warm cache, a switch costs one prefill of the diff plus the new message, and cached-read costs scale with turn count, which switching does not increase - so the dominant agentic cost (cached read + generation) can drop when generation is delegated to a cheaper model. Caveat raised: 'Input tokens: Pay for both models.' (4) Users benchmarked throughput in llama.cpp at 10k context without MTP: 27B ~42 tok/s, 35B ~135 tok/s (~3x, not the 8x one user assumed); with MTP, 27B ~60-70 and 35B ~180-200 tok/s, with one user noting MTP is lossless at temp=0. (5) An independent agentic test (building a collaborative whiteboard on Cloudflare OS) found the MoE models Qwen 3.6-35B and Nemotron 3.5 Lightning 'terrible... couldn't get the job done at all,' while ~30B dense models were 'pretty decent.' (6) Skepticism about the premise: 'we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?' plus a broader Bitter Lesson debate over RL/small-model scaling vs raw parameter scale. (7) The docs never mention 'cache' although the code does. Data caveat: roughly 10 of the top-25 comments shown are off-topic posts from an unrelated thread about Bulgaria/EU politics and carry zero signal about Switchyard.

No aggregate ratings were found for this product during the last harvest.