Building My AI Halloween Witch and Grim Reaper

This year, I’ve been building a Halloween setup with a witch, a Grim Reaper, and a bride, each with their own voice and behavior. The witch can notice visitors and generate remarks about what she sees. The Grim Reaper has speech and sound effects synchronized with his physical animation, and the characters can tell prepared stories together. Behind the display are Raspberry Pis, cameras, a Linux server, OpenAI for image analysis and dialogue, and ElevenLabs for speech. A shared dashboard lets me control the setup from my phone, inspect what the software is doing, and intervene when I need a particular line or announcement.

I've had the witch for a long time, pieced together from Dollar Tree items and then slowly added to over the years. The Grim Reaper was an animatronic I bought that just has gotten old with repeated phrases and everything else. It's a soul sucker, and it's pretty neat, but I decided to do something a little more fancy with it.

The project has grown considerably from the initial experiment of capturing an image and making a witch speak. Once that worked, the questions became more practical. How would she see someone inside a dark tent? How would she avoid greeting the same family repeatedly? How could the Reaper’s new audio follow an existing animatronic without modifying its wiring? And once several characters could speak, what determined whose turn it was? Those details account for much of the work, and they’re what turned the early prototype into something I could operate as a Halloween display.

How the witch sees and addresses visitors

The witch’s interaction begins with a camera image. The system checks for visitors, then uses visible details to generate a short piece of dialogue grounded in the scene. Clothing, costume accessories, visitor counts, and visible gestures can all give her something to acknowledge. The generated text goes to ElevenLabs, which produces speech in her selected witch voice. This is a camera-and-speaker interaction: she can ask questions, but the setup doesn’t listen for spoken answers. It also doesn’t use facial recognition to identify people. When she comments on someone, the intended basis is something observable in the current image.

The first version used a USB webcam and a local person detector, but the actual display was much more difficult than a reasonably lit indoor test. The witch sits in a tent with dark webbing, theatrical mist, a cauldron, and decorative lighting. An early camera test produced an almost black image, yet the model still supplied a costume description. That was a useful failure because it exposed the danger of treating a plausible response as proof that the camera was working. The application needed better handling for dark or unusable images, and the physical setup needed a camera that could actually see the scene.

The later version uses a Wyze camera running Thingino firmware to provide an infrared video feed over the local network. The witch’s Raspberry Pi reads that feed and keeps the latest decoded frame available for processing. Stronger OpenAI vision checks assess whether visitors are present, with repeat behavior governed by the application’s state rather than by each response in isolation. After a greeting, separate confident observations of an empty scene are required before another arrival can trigger her. An uncertain result doesn’t count as someone leaving. That helps prevent a family standing in front of the witch from being greeted over and over whenever lighting, movement, or a missed observation changes the detector’s answer.

Speech generation is streamed so the system can begin preparing a completed line while later dialogue is still being generated. Cached greeting and fallback recordings provide ready audio when a cloud response takes longer, while saved remarks and background ambience give the character activity between encounters. These paths have different dependencies: a personalized remark needs the image-analysis and speech services, while an already prepared recording can play locally. Keeping track of those differences matters when diagnosing silence or delays, because “the witch isn’t talking” can mean several different things.

The dashboard became part of the build

As the behavior expanded, the dashboard became the main way to operate and troubleshoot it. The unified interface runs on the Linux server and provides separate Witch and Bride tabs, with Harvester controls included in the Bride section. From my phone on home Wi-Fi, I can start or stop the witch, stop only her current line, test vision or speech, change the scare level, and issue a prepared movie announcement. The interface uses my usual dark navy background, pale text, ice-blue controls, and violet active indicators, but the layout is driven by practical needs: the controls I might need during an interaction are grouped together and easy to reach.

The Witch controls include exact-text speech, temporary visitor labels, and a scare-level selector.

One of the most useful controls is the plain text box for manual speech. I can type exactly what I want the witch to say and send it directly to ElevenLabs without having a language model rewrite it. That gives me a dependable way to make an announcement or respond to something the automatic system doesn’t understand. The visitor-name controls work through an explicit label and a visible cue: I supply a name along with something that distinguishes the person in the image, and the witch can use that name when the cue clearly matches. The names are temporary, and the feature doesn’t create a facial identity database.

The dashboard also exposes the information needed to understand the pipeline: camera status and preview, recognized visual details, generated dialogue, audio output, and processing times. Timing needs some care here. The arrival of audio bytes, a write to the sound system, and actual sound coming out of a speaker are different events. The software reports what it can measure, which helps locate a delay without pretending those numbers are microphone measurements. Separate vision and voice tests are useful for the same reason: they let me check one part of the system at a time.

How the Grim Reaper follows the animatronic

The Grim Reaper, called Harvester in the software, presented a different integration problem. His existing pressure mat already starts the physical prop, and I wanted the added audio to follow that animation. The eventual solution was to have a camera watch his illuminated eyes. The detector examines a selected region of the image and compares it with calibrated off and lit states. When the eyes illuminate, the application has a visible signal that the prop has activated and can start the corresponding audio. The pressure mat and the prop retain their existing relationship; the added software observes the result.

The activation panel provides off/on calibration, a detection target, and controls for timing and rearming.

That sounds straightforward, but reliable triggering required more than looking for bright pixels. The watched area needs to cover the eyes without including too much unrelated scenery, and the detector needs stable observations so a momentary change doesn’t count as an activation. It also needs to recognize one animation cycle as one event, even though the eyes remain illuminated across many camera frames. The application therefore includes rearming and cycle timing, along with checks intended to distinguish the eye lights from broader changes in illumination. Moving the display beside the deck stairwell required fresh nighttime calibration because the camera angle and background conditions had changed.

The Reaper’s audio also needed a few iterations. At one point, the recordings became longer spoken monologues, but the intended effect depended on more than dialogue. The current pack contains ten prepared 20-second sequences, each combining a short menacing line with a rising vortex effect, release, and decay. During that activation, ordinary background ambience mutes and then fades back afterward. The sequence gives the physical animation a coordinated sound treatment, while prepared audio avoids adding a cloud-generation delay at the moment someone steps on the mat. Nighttime testing included actual mat activations, separately from tests that invoked only the audio endpoint.

Getting the characters to tell stories together

Once the individual characters could speak, the project expanded into stories performed across their voices. The initial library contains five stories with 147 spoken passages, plus separately prepared title introductions. The Linux server owns the scripts, cached recordings, and story director. It sends passages to the appropriate player and tracks their progress, while the Raspberry Pis handle playback. ElevenLabs is used when preparing the recordings; a prepared story doesn’t need a new speech request for every line. The dashboard also accepts Markdown stories, with headings assigning passages to Witch, Bride, or Harvester.

Stories have explicit playback controls and availability status. This capture shows a stopped story and the Witch offline.

Turn-taking became one of the more noticeable performance problems. Early handoffs left several seconds between characters, which made dialogue feel disconnected even when the individual recordings sounded good. Removing a fixed pause, shortening polling intervals, and reusing network connections reduced two measured handoffs from more than three seconds to roughly half a second. Those figures came from observed playback states over the network. A later revision allowed the next character to start slightly before the previous recording ended when the voices used separate physical speakers. Bride and Harvester share one speaker, so their passages remain sequential. The director tracks both outgoing and queued playback so Pause or Stop can cancel a handoff already underway.

Stories also forced an explicit decision about priority. The current behavior gives the operator-started story control over automatic greetings, side chatter, and camera-triggered Harvester audio. Those automatic behaviors remain suspended while the story is manually paused, and background ambience drops to 20 percent of its selected level. Stop or natural completion releases story mode and restores ordinary behavior. This took revisions to get right because several independent features were each behaving reasonably on their own, yet could still interfere with the performance when combined.

Much of what I’ve learned has come from that interaction between software and the physical display. Low-light camera performance, the position of a small watched region, speaker connections, and the silence between two lines can matter as much as the generated dialogue. The dashboard grew alongside those discoveries, giving me a way to inspect the system and make adjustments without opening a terminal every time. The setup now combines visitor-specific witch dialogue, prepared character speech, synchronized Reaper effects, and coordinated stories. The remaining test is how consistently all of those pieces work together through an evening of visitors.