Coping with dumb LLMs using classic ML

As I see it, you need a model you can train quickly so you can do tuning, model selection, and all that.

I have a BERT + SVM + Logistic Regression (for calibration) model that can train 20 models for automatic model selection and calibration in about 3 minutes. I feel like I understand the behavior of it really well.

I've tried fine tuning a BERT for the same task and the shortest model builds take 30 minutes, the training curves make no sense (back in the day I used to be able to train networks with early stopping and get a good one every time) and if I look at arXiv papers it is rare for anyone to have a model selection process with any discipline at all, mainly people use a recipe that sorta-kinda seemed to work in some other paper. People scoff at you if you ask the engineering-oriented question "What training procedure can I use to get a good model consistently?"

Because of that I like classical ML.

I spent a week learning enough ML to design a recommender system that worked well with my company's use case. I knew enough linear algebra to determine that collaborative filtering with some specifically chosen dimensionality reduction and text vectorization algorithms as well as a strategy for scaling the models across multiple databases would work well for us. The solution was tailored specifically to the type of data we were working with.

When I presented the proposal, nobody read it and the meeting immediately turned to the vp of engineering and the ceo discussing neural networks and some other ML system that they had read about on HN the day before. When I tried to bring collaborative filtering up again, the VP said "I don't know what that is", so obviously he hadn't read the doc that I was assigned to write over the last week

LLMs are inherently bad at this due to tokenization, scaling, and lack of training on the task. Anthropic’s computer use feature has a specialized model for pixel-counting: > Training Claude to count pixels accurately was critical. Without this skill, the model finds it difficult to give mouse commands. [1] For a VLM trained on identifying bounding boxes, check out PaliGemma [2]

You may also be able to get the computer use API to draw bounding boxes if the costs make sense.

That said, I think the correct solution is likely to use a non-VLM to draw bounding boxes. Depends on the dataset and problem.

1. https://www.anthropic.com/news/developing-computer-use 2. https://huggingface.co/blog/paligemma

Maybe still worth it to separate the tasks, and use a traditional text detection model to find bounding boxes, then crop the images. In a second stage, send those cropped samples to the higher-power LLMs to do the actual text extraction, and don't worry about them for bounding boxes at all.

There are some VLLMs that seem to be specifically trained to do bounding box detection (Moondream comes to mind as one that advertises this?), but in general I wouldn't be surprised if none of them work as well as traditional methods.

AFAIK none of those models have been trained to produce bounding boxes. On the other hand Gemini Pro has, so it may be worth looking at for your use case:

https://simonwillison.net/2024/Aug/26/gemini-bounding-box-vi...

I am doing OCR on hundreds of PDFs using AWS Textract. It requires me to convert each page of the pdf to an image and then analyze the image and it works good for converting to markdown format (which requires custom code). I want to try using some vision models and compare how they do, for example Phi-3.5-vision-instruct.

1. You need to look into the OCR-specific literature of DL (e.g. udop) or segmentation-based (e.g. segment-anything)

2. BigTech and SmallTech train their fancy bounding box / detection models on large datasets that have been built using classical detectors and a ton of manual curation

> they failed miserably at finding bounding boxes, usually just making up random coordinates.

This makes sense to me. These LLMs likely have no statistics about the spatial relationships of tokens in a 2D raster space.

Gemini 2 can purportedly do this, you can test it with the Spatial Understanding Starter App inside AI Studio. Only caveat is that it's not production ready yet.

I think people have had success with using PaliGemma for this. The computer use type use cases probably use fine tuned versions of LLMs for their use cases rather than the base ones.

I feel the same way about programming, but there are plenty of people that don't enjoy it.

I recently was chatting with my friend that wanted to automate one of his tasks by writing a python script with AI -> because all the influencers said it was "so easy" and "no programming knowledge" required.

That might have been the single funniest piece of code I have seen in a long time. Didn't install the dependencies, didn't fill in the Twitter API key, instead of searching for a keyword on Twitter it just looked up 3 random accounts, 25 functions on like 120 lines of code?

Also, the line numbers in the errors weren't helpful because the whole thing lived in Windows notepad. That was a flagship AI and a (in my opinion) capable human not being able to assemble a simple script.

TBH I'm not sure its better, but the decision tree structure is pretty handy for problem exploration

(However 'better' might be defined, I care more about the precision / recall tradeoff)

This resonates with my experience. Use LLMs for feature engineering, then use traditional ML for your inference models.

It seems like a really easy way to overfit your model to your data, even while using LLMs.

That's how we all solve problems. If this was novel, it would be a paper rather than a blog post.

The meta-strategy of combining LLM and non-LLM techniques is going to be key for getting good results for some time.

> he just changed the shape of the problem into one that was familiar to him

that's a classic strategy to solve problems

Yes that's a great idea, and maybe something I would try next in this series.

With or without the bug it's a horid prompt. Prompts work best when they resemble content LLMs have in their training data. People use first and second far more often then LHS and RHS when talking about options. First or second, 1 or 2, a or b or neither.

LLMs are narrative machines. They make up stories which often make sense.

This is a copy/pasted typo, the real prompt begins

> Which of these furniture products is more relevant to the furniture e-commerce search query:

Fixed in the post. Thanks

And like the philosophers stone it does not exist. Remember the "Map vs Territory" discussion: you cannot have generic maps, only maps specialized for a purpose.

The amount of hallucination I get when trying to write code is amazing. I mean it can get the core concepts of language, can create structure/algo. But it often makes up objects/values when I ask questions. Exampe: It suggested TextLayoutResult.size - which is Int value. I asked if it is width and height. And it wrote it has size.height and also size.width. Which it does not. I am now writing production code and also evaluating the LLMs, that our management thinks will save us shit load of time. We will get there sometimes, but the push from management is not compatible with the state of the LLMs. (I use Claude 3.5 sonnet now, as it is also built in some of the "AI IDEs".)

... I just realized that I would be waking up just to go back to resting.