The Bug
AutoPost generates Instagram content automatically — captions, carousels, hashtags — for food brands. One part of that pipeline was broken in a way that looked fine on the surface but was factually wrong: every post got assigned a product photo at random.
A post about a Chicken Tikka Rice Bowl could end up showing a photo of a Rajma Chicken Salad. The caption is specific. The photo is lying. For a food brand where the whole point is making customers want to order that dish, this is a real problem — brand trust, customer confusion, and in the worst case, someone ordering something that looks nothing like what showed up.
The fix needed to be context-aware: the system had to understand what a post is actually about before choosing a photo.
Why Random Happened
The original image selection was one line — pick from the available photo library, return one. There was no semantic link between the post content and the photo. The library existed. Posts got generated. A photo got attached. Done.
This works fine when you have one product and one photo. It breaks immediately when a brand has 15+ dishes, each with its own photo (and some without), and the content pipeline is generating posts about specific dishes.
What I Built
Step 1: Tag the photo library once
Every product photo gets run through Claude's vision capability to generate structured tags: what dish it is, the main ingredients visible, the food category (protein bowl, salad, wrap, etc.), and any distinguishing visual characteristics.
The critical detail here is grounding the tags against the client's actual menu. If the menu calls it "Rajma Chicken Salad," the tag uses that exact phrase — not "bean salad" or "mixed salad" or whatever a generic vision model might default to. This matters because the posts are also generated from menu data, so the vocabulary has to match. If the post says "Rajma Chicken Salad" and the photo is tagged "bean and chicken salad," a naive string match fails even though they're the same dish.
This tagging happens once per photo, gets stored, and doesn't run again unless the library changes. It's not a per-post cost.
Step 2: Score photos against post content
For every new post, the system scores each tagged photo against the post content. The scoring looks at dish name overlap, ingredient mentions, and category alignment. A post about a rajma salad scores the rajma salad photo high and everything else low.
This produces a ranked list with scores. The top result wins — but only if it clears a confidence threshold. This is the part that matters for correctness.
Step 3: Fall back honestly when there's no good match
If the best-scoring photo doesn't clear the threshold, the system falls back: either to a generic brand photo or to the old random behavior (configurable per client). The key is it doesn't fake a match.
This is what fixed the Chicken Tikka Rice Bowl case. That dish has no photo in GigaFit Meals' current library. The old system would grab a random photo and attach it. The new system scores all available photos, finds that none of them are about chicken tikka rice bowl, and falls back cleanly — a generic photo goes on that post instead of a photo of a completely different dish.
Failing honestly is better than succeeding incorrectly.
Extending It to Carousels
Carousels made this more interesting. A carousel post for a meal prep brand often features multiple dishes — slide 1 is the rajma salad, slide 2 is a grilled chicken bowl, slide 3 is a wrap. The old behavior: one photo, repeated across all slides, or random photos per slide with no coherence.
The fix runs the same matching logic per slide. Each slide's content gets scored against the photo library independently, and each slide gets its own best-match photo. If slide 2 is about a grilled chicken bowl and there's a photo of that dish in the library, slide 2 shows that photo. Slide 1 gets the rajma salad photo. Slide 3 gets whatever matches the wrap content, or falls back if there's no match.
This required the carousel generation to expose per-slide content to the image matcher rather than treating the whole carousel as one unit. Small structural change, meaningful output difference.
Test Results on GigaFit Meals' Real Content
Ran this end-to-end on actual Instagram content for GigaFit Meals.
Case 1: Dish with a photo in the library Post about Rajma Chicken Salad → system scores the rajma salad photo highest → post shows the actual rajma salad photo. Correct.
Case 2: Dish with no photo in the library Post about Chicken Tikka Rice Bowl → no photo in the library for that dish → system finds no confident match → falls back to a generic photo. The post stops lying about what the dish looks like.
Case 3: Multi-dish carousel Carousel with three different dishes → each slide independently matched → each slide shows the right photo for that slide's dish. Previously this was either one photo repeated three times or three random photos.
The Architecture in Plain Terms
The pipeline now has three stages where it used to have one:
- Library indexing — runs once per photo, produces structured tags grounded in menu vocabulary
- Post-time scoring — runs per post (or per slide), compares post content against all tagged photos, returns ranked matches with confidence scores
- Selection with fallback — takes the top match if it clears threshold, otherwise falls back to a configured default
The vision step (Claude tagging the photos) is the only expensive part, and it only runs once per photo. The scoring at post-generation time is cheap — it's matching text against stored tags.
What This Fixes in Practice
Before this change, AutoPost's image selection was decorative — it attached a photo to look complete but the photo had no relationship to the post content. For a personal meal delivery brand where visual accuracy is part of the product (customers are literally deciding what to order based on these posts), random photo assignment was a quiet but real failure.
The matcher isn't perfect. If two dishes look visually similar and are described similarly, the scoring can return the wrong one. The confidence threshold is tunable, and false positives are still possible — but the floor is much higher than random, and the fallback means the system fails safe when it's uncertain.
This is the kind of bug that's easy to miss because the output looks complete. A post with a photo attached looks like a finished post. The problem only becomes visible when you look at whether the photo is correct — which requires knowing what the post is actually about.