https://sdk.vercel.ai/docs/introduction
It uses zod for types and validation, I've loved using it to make my apps swap between models easily.
Their docs (incl. getting started tutorials) are content marketing for the platform services, needlessly pushing new users into more complex and possibly unnecessary direction in the name of user acquisition.
(I have the same beef with NextJs/Vercel and MongoDB).
Some time ago I built a rather thin wrapper for LLMs (multi provider incl local, templates, tools, rag, etc...) for myself. Sensible API, small so it's easy to maintain, and as a bonus, no platform marketing shoved in my face.
I keep an eye on what the LangChain ecosystem's doing, and so far the benefits ain't really there (for me, YMMV).
Either way I guess.
I would have thought this was impossible, I contribute to llama.cpp and there's an awful lot of per-model ugliness to make things work, even just in terms of "get it the tool calls in the form it expects."
cries at the Phi-4 PR in the other window that I'm still working on, and discovering new things, 4 weeks later
I’d guess it supports a small set of popular HTTP APIs (particularly, its very common for self-hosting LLM toolkits—not the per-model reference implementations or low-level console frontends—to present an OpenAI compatible API), so you could support a very wide range of local models, through a variety of toolkits, just by supporting the OpenAI API with configurable endpoint addresses.
That's not to say won't need to tweak things when you cut down to smaller models, there are always trade offs swapping models.
Its Apache 2.0 licensed
How is this allowed/possible in npm? Don't they have mandatory namespaces?
> How is this allowed/possible in npm? Don't they have mandatory namespaces?
No, scopes (i.e., namespaces) aren’t mandatory for public packages on npm. If a name is available, it’s yours. Sometimes, some folks have published packages that are either empty or not used, so you can reach out to them to ask them to pass it on to you. At least a few times, I’ve had someone reach out to me asking for a package I had published years ago that I did nothing with and I passed it on to them.
That works for me reliably. I've had some issues with running into max_token constraints but that was usually on me because I had let it process a large list in one inference call, which would have resulted in very large outputs.
We're using gemini JSON mode in production applications with both `google-generativeai` and `langchain` without issues.
export const
function
type
return
etc
This makes scanning through the code really hard because your eye has to jump horizontally.
options: [
Omit<ChatCompletionParseParams, 'model'> & { model: Model },
Omit<ChatCompletionParseParams, 'model'> & { model: Model },
],
type Options = Omit<ChatCompletionParseParams, 'model'> & { model: Model }
options: [Options, Options]