A GPT-4 Program a Day Keeps the Bugs Away
I’ve been exploring what GPT-4 and other AI tools are capable of over the past few months when it comes to software development.
I’ve mostly been using ChatGPT+, Bing Chat – I had tried to use some of the other hyped up models for programming purposes, but nothing I’ve found lets me build software from scratch (while writing minimal code myself) as well, and so these days I mostly stick to those.
Since the AI space changes seemingly daily, I’m sure this will be laughable and incorrect shortly if it’s not already.
After being impressed with some tests I threw at Bing and seeing the potential for what it’s capable of in terms of coding, I decided to try to write a program with it each day whenever I had some extra time in the morning.
What are the Rules?
I didn’t set any firm rules, but tried to follow these guidelines:
Size doesn’t matter - The program can be large, small or anything in-between. It might be a userscript or a bash script to automate something simple or even a fully working video game engine.
Done in a day - The big goal I had in mind is to have a fully working project by the end of a day. On days I had more free time and an interesting idea to test, like the game engine, I may spend hours on a project. For the smaller projects like a bash script, I might be done in 10 minutes.
AI does everything - I should build prompts to let the AI make as many decisions as possible and step in when it fails or can’t do something correctly. This includes pattern and library selection, project structure, deployment, writing the actual code, and even helping flesh out the idea itself.
No (or very little) coding - I should not write any code myself. Surprisingly, this stayed true (barring little syntax errors) for nearly all the projects I’ve done. It is easier to just tell it how to modify its code how you’d like.
A big benefit of going through this was gaining a good understanding of what kind of complexities GPT-4 can handle and what kinds of projects I can (now) do in a day.
I was shocked by the productivity gain and how interesting side projects I would have never started on due to time commitments, I could now finish in a few hours.
Some Examples
All the examples below and all the AI coding projects I do now are using Bing Chat creative mode. It seems to have the right mix of knowledge, imagination and desire to try to answer queries that other GPT4 model’s system prompts won’t allow them to.
For example, if you ask ChatGPT+ to build an entire game engine, it will likely balk and say that’s too complicated for it, while Bing Chat will warn you that’s tough for it to do but still try to work something out with you. This opens the door to much more complicated ‘from scratch’ AI-built applications.
I’ve omitted most of the ‘simple’ projects I’ve done here, but Bing is really great at that and I encourage you to try that out to start.
didn’t even try…
Projects
(click the carats to expand)
Play Mountain Climber here!
See a video here The most complex program I’ve been able to make with GPT (in a day) so far is a three.js based adventure game engine. You can use it to make multiple choice adventure games with multiple paths and scenes, like a choose your own adventure book. The engine takes a JSON file that describes entire games including assets, all scenes/choices and arbitrary animation code executed in the background for each scene. The repo has a schema file that describes the structure of the ‘game cartridge’. New games can be made by giving this schema to Bing and describing your game. Getting it to create animation code that did anything of note was challenging and frustrating, so all of the examples I made I just had it rotate the camera around a skybox and generated those using Blockade Labs and prompts for each scene. I was really impressed it was able to build something like this so quickly. While this took a long time, almost all the time was spent adding features, working through bugs, and fighting (unsuccessfully) with the animation code concept. Here’s the prompt used initially… Help me build a framework in three.js to power simple text adventure games. The framework should easily consume a structured text file that defines a game’s settings, scenes, user choices, and everything else necessary to run a game. The structured file, or some other means if you have a better idea, should also contain a section for animation code that will be run alongside the different scenes. The intent here is a user can build a text file with scenes, choices, etc and the animation code, to relatively easily make a text adventure game with graphical elements. Let me know if you have any questions before you begin, or, go ahead and get started – Thanks!A Choose Your Own Adventure Game engine with JSON cartridge support
Use EchoSculpt here! - Use the mouse or your fingers to rotate the echosculpt. Click it in different areas to produce different sounds. See below for more. This was my favorite experience coding with Bing. I asked GPT to imagine a bunch of new musical instruments that don’t exist and describe them in detail. Then, I asked it to pick one to make a simulation of using Three.js. EchoSculpt: The EchoSculpt is a sculptural musical instrument that generates sound based on the way it is touched and manipulated. It is a large, geometric, and multi-faceted structure made from a combination of metal, glass, and advanced acoustic materials. Each facet of the sculpture produces a unique sound when struck, rubbed, or otherwise manipulated, allowing for an infinite number of sonic possibilities. This took several hours, but Bing wrote all the code. Bing also generated prompts for other AI tools to generate the background and model files. It was able to figure out the basics of the implementation and most of the time was spent working through bugs and changing sounds to differ more. I also had it add some of the visual features like the background, the color lights, the color changing model, etc.Imagining a new musical instrument and simulating it in Three.js
This is one I never got properly working due to GPT-4s relatively poor understanding of space and graphics. I will probably use this as my default AI test in the future when testing new models. The text conversion part of the script worked, but getting it to figure out how to output a file with the margins working correctly did not. The characters weren’t spaced evenly like in the above image and I wasn’t able to get Bing to fix the issues. I realize now I could probably ask it to add a space at every character and work around the margin issues, but we didn’t come up with that idea at the time.Python script to create Boustrophedon text
One of my first projects was asking Bing to improve itself by building a userscript to add a copy button to codeblocks. This is outdated now as that’s been added in the default UI, but it was helpful before that was added. Along with copying multiple codeblocks in a single copy, it also was able to add the feature of automatically adjusting the clipboard for long pastes. In previous versions, Bing Chat would not allow more than 2000 characters in the textarea for chat. This userscript adjusts the clipboard to remove the part you pasted, so you can easily paste multiple prompts without going back to your text editor. It was not able to parse its own DOM and I had to give it a selector. I didn’t try sharing the HTML directly which may have worked.A helper script for Bing Chat GPT-4
A Tomogotchi like Video Game
I wrote an article about this one, so I won’t detail it here. This was an early test for me with mixed, but impressive results considering my very low expectations at the time.
Prompts & Process
I haven’t seen a ton of people using or talking about using GPT-4 this way. Products like Github copilot and the like (which seem to be what most people use and discuss) keep you in the driver seat and uses the AI to get help, ideas, or autocomplete code you’re writing.
I know this style of development is happening and the glut of ChatGPT wrapper apps and news like Valve banning games using AI assets from Steam imply that as well. But, the professional developer community seems disinterested in the ground-up style AI development process I’ve used here.
That disinterest does make sense too – it is both a threat to the value of coder’s labor and is a very different way of working. It’s more akin to telling someone to write a program for you and guiding/correcting them than it is feeling empowered in a familiar process with a new tool.
An inordinate amount of time is spent, almost always, coaching GPT-4 to not use hallucinated libraries, fix obvious syntax errors in its code, asking it to continue code outputs – I won’t lie, it is pretty frustrating today.
Still – If this is the V1/Model-T of LLMs and AI programming, I can’t help but bet on the most efficient software writing process of the future being letting the AI take the leading role while developers guide it.
The future is inside the computer??
For most simple things, you can simply ask Bing, “Write a Python script that takes a folder of jpgs and converts them to pngs” or the like.
Longer projects require quite a bit more effort and I’ve found spending time defining things before starting saves a lot of time in the end (shocking developer revelation I know). My flow with these longer daily projects has looked like this:
Steps for Writing a Program with Bing
- Define project and desired end result
- Can Bing help? Consider asking for it to decide on/add features, color palettes, rule sets, objects, etc.
- Consider asking it to #optimize your idea for your audience, goal, or other
- Define architecture and folder/file structure
- Ask Bing to define all the libraries and tools we’ll need to fully complete the project
- Make sure to tell it to ask if it has questions before it begins
- Bing loves to hallucinate libraries that don’t exist, so check all parts of your project are real and accounted for here.
- Generate project description and setup/development step list
- These will be very useful for longer projects where we will run out of our 30 prompt limit for Bing Chat. You can feed this back into a new session along with the already written code for bigger projects.
- Asking it for project setup steps will usually be skipped over if you don’t ask explicitly
- Generate list and description of assets
- If the project has external assets like images, models, fonts, etc, ask for a list of those up front
- Begin working through setup/development steps
- It’s good to stop and test when able and work through any bugs. You can also ask Bing to check for bugs or other issues with some success.
- For long codeblocks that get cutoff, you can usually say ‘continue’ and it will continue the output from where it left off.
- There will likely be a lot of bugs for more complex projects, but I’ve found it can fix about 1/2 of them once it knows the behavior exists. The other 1/2 are usually resolved by Bing once it knows why the bug is occurring (which I have to figure out)
What’s Next?
I encourage you to explore building something with GPT-4 in this way and see if you find the same potential for productivity gains that I have.
The biggest help I’ve found (kind of obviously) is with languages/libraries/concepts I’m not familiar with. If you could just sit down and blast out code for something, you’ll probably not enjoy working with GPT-4 to code it (unless it’s simple and verbose) so try something you’re unfamiliar with, like I did with game development.
Next, I will likely break my ‘guideline’ of not working on a project more than a single day and see how far I can take the Adventure Game Engine mentioned earlier in this post. There’s no reason (with some API integrations) it couldn’t generate ALL game assets and content on demand with input from the user, and that sounds interesting to explore and a great demo of FULL AI driven and built software.
Thanks for reading!