Ask Joget AI

Using Custom Large Language Models

Pre-requisites for Custom Models in Joget AI Designer

The Joget AI Designer is tested extensively with OpenAI and Anthropic models, enabling seamless app generation. However, it can work with any large language model (LLM) that supports OpenAI-compatible APIs.

Here are some key requirements and recommendations for using custom models effectively:

Key Requirements for using LLMs

Use Large, Proficient LLMs.

Reasoning and Structure: The LLM should handle complex prompts well and generate outputs that maintain proper YAML or JSON formatting.
Size Considerations: Small models often lack reasoning power, which can lead to failed outputs if the prompts are too complex. Use models with proven proficiency in reasoning.

Multi-Modal Support for Image Features.

To enable image recognition features, the LLM should support multi-modal inputs.

OpenAI-Compatible Server for Integration.

Requirement: Any custom LLM must run on an OpenAI-compatible server for consistent integration with the Joget AI Designer.
Recommendation: Use frameworks like vLLM, which offer OpenAI-compatible servers out of the box.
Reference: For more details, see vLLM OpenAI-compatible server documentation.

Handling Proprietary LLMs.

Wrapper Solution: For proprietary models without direct OpenAI compatibility, build a simple FastAPI application wrapper on top of the existing LLM API endpoint to create an OpenAI-compatible server.

Automated Error Correction and Feedback Mechanism

Feedback System: The Joget AI Designer has a feedback mechanism that sends corrections to the LLM if the output doesn't make sense or if there’s an error in validation.

Guardrails for Smaller LLMs:
Guardrails rarely fail, but complex prompts may exhaust the LLM’s maximum calls if the model isn’t proficient. In such cases, try:

Using a more robust LLM.
Simplifying or clarifying the application description within the prompt.

Manual Refinement:

A “refine” button allows users to make corrections using the LLM without needing manual intervention.
A “human-in-the-loop” model is not yet implemented, so direct human corrections for malformed outputs aren't part of the current design.

Example OpenAI Compatible Server Wrapper

(Only for reference, actual implementation may differ)

https://platform.openai.com/docs/api-reference/chat/create

LLM Wrapper

import uuid
from datetime import datetime
from typing import List, Optional, Union


import httpx


from pydantic import BaseModel


from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse




app = FastAPI()


# Replace these with your actual LLM API keys and endpoint
YOUR_API_KEY = "your_api_key"
YOUR_API_ENDPOINT = "https://api.llm.com/v1/complete"




class Message(BaseModel):
   role: str
   content: str




# OpenAI-style request schema
class OpenAIRequest(BaseModel):
   model: str
   messages: List[Message]
   max_tokens: Optional[int] = 100
   temperature: Optional[float] = 0.7
   top_p: Optional[float] = 1.0
   n: Optional[int] = 1
   stop: Optional[list] = None




# OpenAI-style response schema
class Choice(BaseModel):
   index: int
   message: Message
   logprobs: Optional[Union[dict, None]] = None
   finish_reason: str




class CompletionTokensDetails(BaseModel):
   reasoning_tokens: int
   accepted_prediction_tokens: int
   rejected_prediction_tokens: int




class Usage(BaseModel):
   prompt_tokens: int
   completion_tokens: int
   total_tokens: int
   completion_tokens_details: CompletionTokensDetails




class OpenAIChatResponse(BaseModel):
   id: str
   object: str
   created: int
   model: str
   system_fingerprint: Optional[str] = None
   choices: List[Choice]
   usage: Usage




@app.post("/v1/chat/completions", response_model=OpenAIChatResponse)
async def create_completion(request: OpenAIRequest):
   # Translate OpenAI request to LLM-compatible format
   llm_payload = {
       "prompt": request.messages,
       "model": request.model,
       "max_tokens_to_sample": request.max_tokens,
       "temperature": request.temperature,
       "stop_sequences": request.stop or [],
   }


   headers = {
   "Authorization": f"Bearer {YOUR_API_KEY}",  # or use "x-api-key": YOUR_API_KEY
   }


   # Make the request to LLM API
   async with httpx.AsyncClient() as client:
       response = await client.post(
           YOUR_API_ENDPOINT, json=llm_payload, headers=headers
       )


   if response.status_code != 200:
       raise HTTPException(
           status_code=response.status_code, detail="Error from LLM API"
       )


   llm_response = response.json()


   # Map LLM's response back to OpenAI-compatible response
   openai_response = OpenAIChatResponse(
       id=llm_response.get("id", uuid.uuid4().__str__()),
       object="chat.completion",
       created=int(datetime.utcnow().timestamp()),
       model=request.model,
       system_fingerprint="random",
       choices=[
           Choice(
               index=0,
               message=Message(role="assistant", content=llm_response.get("response")),
               logprobs=None,
               finish_reason="stop",
           )
       ],
       usage=Usage(
           prompt_tokens=0,
           completion_tokens=0,
           total_tokens=0,
           completion_tokens_details=CompletionTokensDetails(
               reasoning_tokens=0,
               accepted_prediction_tokens=0,
               rejected_prediction_tokens=0,
           ),
       ),
   )


   return JSONResponse(content=openai_response.model_dump(), status_code=200)

Using/Testing LLM Wrapper

from openai import OpenAI


# Note: We don't support all OpenAI client arguments and may ignore some.
client = OpenAI(
   # Replace the URL if deploying your app remotely
   # (e.g., on Anyscale or KubeRay).
   # NOTICE How the /chat/completions is not added here. (Auto added by OpenAI
     internally)
   base_url="http://localhost:8000/v1/",
   api_key="NOT A REAL KEY",
)
chat_completion = client.chat.completions.create(
   model="YOUR MODEL",
   messages=[
       {"role": "system", "content": "You are a helpful assistant."},
       {
           "role": "user",
           "content": "Hey!",
       },
   ],
   temperature=1,
   max_tokens=2000,
   extra_body={"stop_token_ids": [128009]},  ## This is for model specific inputs
)


choices = chat_completion.choices[0]
choices = dict(choices)
text = choices["message"]["content"]


print(text)