Can ChatGPT Write Code? Let's Experiment to See If It Can Replace Developers Completely

It seems that literally everyone has heard about artificial intelligence over the past year, it became an integral part of our lives and many companies are integrating AI tools into their software products. The names ChatGPT and Midjourney have already become common nouns, as AI is the number one trend in 2023.
Until the end of 2022, everyone was assured that these technologies had no risk for creative and intellectual occupations. However, current AI capabilities seem so shocking, that everyone has expressed their fears at some point. The results of AI graphic and musical activities are beginning to be regulated by laws. But what about software development? Will developers lose their jobs because of neural engines? Well, in a nutshell, they will not, but their work will change significantly.
In this article we did an experiment and tried to create a Google Chrome extension using ChatGPT to answer whether ChatGPT can write code, how good it is at generating code, what will happen to software development and how ChatGPT can be useful in development.

What is ChatGPT?

ChatGPT is a natural language processing algorithm from OpenAI, based on artificial intelligence technology, that allows users to communicate with the chatbot as if it was a real person. The AI model was trained by Reinforcement Learning from Human Feedback (RLHF).
The ChatGPT developers discussed some limitations of the tool:

• ChatGPT writes responses that appear to be true in some cases, but they are all fake, and this task is difficult for the following reasons:
  • Reinforcement learning (RL) does not involve outwardly indicating truth and falsehood.
  • Forcing the model to be cautious is a bad idea, because that way it may not answer questions to which it knows the answers.
  • The neural network's answer depends on the knowledge base on which it was trained, not on the knowledge of a particular person, so controlling the model's training can lead to strange results.
• ChatGPT is sensitive to changes in query wording or repeated attempts to enter the same query. For example, the model may not know the answer to a question, but with a small change it will be able to give the correct answer.
• Very often the model is verbose and uses many cliched phrases due to the fact that it was important to the AI-instructors that the neural network give long and exhaustive answers, as well as known over-optimization problems.
• The perfect scenario for a neural network to work with a user is to ask clarifying questions of the user to form the most accurate answer. However, at the moment, algorithms try to guess the context implied by the user.
• During the training of ChatGPT, much effort was put into ensuring that the model refuses to respond to inappropriate queries and harmful instructions. However, it continues to respond to them in a biased manner because controlled learning is bad for the algorithm. To address this problem, OpenAI uses a moderating API that allows it to warn users of unsafe content as well as block it. However, at the moment, OpenAI developers expect such protection to have false alarms.

How to get ChatGPT to write code?

• Using OpenAI’s chatbot directly. OpenAI created a website and mobile apps where users can write prompts to the neural network: you can choose your preferable language, define the role that chatbot would do, make necessary requirements and tasks, and also ask for detecting and fixing errors in code. You can get direct access to it on the OpenAI website as well as through the iOS app. This is the method we use for this article.
• GitHub Copilot X. In 2023 GitHub partnered with OpenAI to launch the GitHub Copilot X tool, which allows you to write code directly into GitHub. It works on the GPT-4 model, which is the most powerful model at the moment, and can write entire functions, explain code in natural language, fix bugs, write unit tests for code fragments and many other things that were shown in the Copilot X announcement. Copilot is also available as an IDE extension for Visual Studio, VS Code, JetBrains IDE and others. Some professionals think that using GitHub Copilot X is more suitable than usual ChatGPT, because it is made specifically for software development and programming.
• CodeGPT is an extension for VS Code IDE using ChatGPT to write code, in particular GPT-3.5 and GPT-4 (this requires API access). Like Copilot X, it makes code on demand, explains code snippets, does refactoring and compiles documentation. In general, it can handle everyday tasks.

Experiment with using ChatGPT to write code

ChatGPT is a powerful tool that is used in many different areas. We wanted to understand how to write code with ChatGPT, and how well it handles this, so we conducted an experiment and tried to create a simple extension for Google Chrome browser. At that time, the newest version ChatGPT (GPT-4) was already out, and it was interesting to see its attempts at writing code.

Initial prompt

The first thing we did is to set the context for ChatGPT: we defined a role for the neural network, which it should perform, and described the task to complete. We wanted to create a simple browser extension to manage several kinds of tasks: work, personal and recurring. We also set the interface layout, namely where each block of tasks should be placed, and we described the extension functionality, such as creating, editing and deleting tasks. Finally, the initial prompt looked like this:

As a result, ChatGPT wrote HTML and CSS code for the extension. We were not particularly surprised by this, because this is not our first AI-using project to use AI, but the code worked and that was good. The neural network also created the manifest.json file for us, which contained the technical and common descriptions of the extension, version number and permissions. This is a necessary file for the extension to work, but developers are usually not that keen on making tech documentation.
At that moment, the chatbot helped us with one of the problems that often occurs at the beginning of work on different projects – the «blank slate» problem. Beginner developers don’t know where to start very often and spend a long time trying to figure out how to make the initial version of the product work. Sometimes even experienced developers have this kind of problem, and ChatGPT usually does a great job.
At this point the result looks like this:

Feature development

Despite the fact that at the very beginning ChatGPT produced HTML and CSS code, the neural network did not generate JS code that processes events, so the requested functionality was missing. ChatGPT was probably interrupted by running out of a certain amount of tokens or symbols and we should have asked to continue the generating process and then we would have received the missing piece of code.
After several attempts we started to fix bugs with ChatGPT and provide a more detailed description of the required functionality. GPT-4 was carefully correcting and supplementing the code, and after some iterations we finally got the code that worked decently.
While ChatGPT was creating JS code for us, we also found out that the accuracy and detail of the prompt is essential from the very beginning. Accurate queries help you spend less time generating code that meets your goals.
As a result, we got a simply designed Google Chrome extension, where users can add, delete and edit work, personal and recurring tasks and mark them if they are done. However, there are some problems that are impossible using ChatGPT as a full replacement of developers.

Challenges in working with ChatGPT

While working with ChatGPT, we noticed several important points that are worth considering in the future.

Code quality

The quality of the initial solutions that GPT-4 offered was comparable to the level of a diligent first-year programming student: the entities were weakly dependent on abstractions, which meant that one of the main Object-Oriented Programming Concepts was not followed properly, and there was a lot of repetition in the code.
ChatGPT wrote code for an extension that locally saved an HTML code with tasks, buttons, and other design elements, which is an unwise decision. A neural network did it as straightforwardly as possible, even though there are better trivial solutions like saving tasks in a json file.

However, if you ask ChatGPT to rewrite code in another way, it would do it properly. Moreover, we noticed different levels of quality of generated code when we were using neural networks in other programming tasks. Perhaps the model was trained on an uneven dataset and there were no good examples of code for some tasks.
At this point GitHub Copilot X probably would have performed better because it is designed specifically for programming. Nevertheless, GPT-4, a non-specialized tool, did a pretty good job.

Interface description

There is an assumption that neural networks don’t have so many examples of linking a verbal description of the software interface to generating code. That’s why making prompts for creating code for complex interface structures is a bad idea. For this purpose, it would be more efficient to use the capabilities of Figma which has recently launched a «developer mode» in beta testing that allows you to create interfaces and get accurate CSS code.

Context loss with a neural network

Problems with the interface didn’t end here. When we got the necessary functionality, we decided to work on the front-end of the extension. This is where the biggest disappointment awaited using ChatGPT to write code. When asked to make the design a little different, the neural network generated new HTML code but didn’t link it to the existing JS code. Fixing bugs started a total disaster. Probably at some point ChatGPT lost context of the conversation and didn’t use the initial requirements to generate new versions of code. When we tried to fix one problem, the neural network made others. And it kept going in a circle.
At some point editing code in this way became a torture, so we asked the neural network to output the entire code with all previous requirements we made. It provided a code with a totally different structure without our requirements. As a result, we did the final result by ourselves based on the generated results.

Results

After four evenings of work, we came up with this result:

Despite all the problems that arose along the way, it is important to remember how we used ChatGPT to write code. Before the last code regeneration we didn’t pay almost any attention to how the code worked and trusted blindly to everything the neural network generated. We put all the smart work on GPT-4 and only checked the resulting code ourselves. For such an irresponsible approach, when all the work was done without much concentration, the result was pretty good.
However, despite the small amount of intellectual and mental effort spent, it is important to keep in mind the amount of time spent to get the solution. We spent a lot of it when we tried to get a working version of a product again. At that point, we think that we should have specified the requirements, reset the context for ChatGPT and started again. It would have saved us a lot of time.

Tips on how to write code with ChatGPT

After the experiment, we have a few tips on how to write code in ChatGPT:

• Write accurate and detailed prompts. Imagine that you give an assignment to a human developer. You need to provide them with enough information to do the task correctly. Otherwise they will ask a lot of questions or write what you didn’t want. Decide what your program or function would do, what you want to include in your code, what parameters you want to send, and more.
  A little later we used a recommendation from other developers that helps to create a detailed product description to use for initial prompt. This provides a better initial result with required functionality and eliminates regenerations that cause problems. We asked GPT-4 this thing:
  I need to develop step-by-step detailed technical specifications for a [product]. Please ask as many questions about the product we would like to build to describe it as precisely as possible. Step-by-step ask questions about each screen, each element.
• Use ChatGPT for finding information about resources and libraries. A big part of modern programming is finding and choosing the right libraries. And if it's also free, that's even better. As you know, programming is not only about writing code, it's also about understanding how to interact with different systems and how to integrate the necessary components. ChatGPT helps with this analysis and provides you with a list of options for your project to go with.
• Ask ChatGPT for code examples. ChatGPT writes code well for some small functions and tasks. However, it won't work out well on a big product even though the generator will be able to create a solution for a necessary or similar problem, which you can adapt and add in your code.

Conclusion

Based on the results of our experiment, we can finally answer the questions we asked at the beginning:

• Can ChatGPT replace developers? No, it can’t. At least, not yet. Neural networks can’t provide a complete solution for your project goals, even after a long dialogue with a chat bot. What’s more, the AI doesn’t write quality code for complicated tasks, and the number of errors will increase as the project grows. But now developers must know how to write code with ChatGPT to successfully delegate small tasks to the AI.

• Can ChatGPT write code? Yes, it can, but with a small nuance: it can write small but efficient pieces of code, not a complex solution. The creation of the basic structure and the implementation of small tasks can be delegated to AI and it will do that effectively.

• How is ChatGPT useful in software development? This tool is useful in many ways:

  • It is useful in studying programming because ChatGPT writes code and shows background information on topics and questions. Thereby, AI makes life easier for entry-level programmers.
  • It can help to start the project from scratch, especially if you don’t know how to do it.
  • Developers get rid of a lot of routine tasks. As mentioned above, ChatGPT handles some small tasks quite effectively. The developer has to deal with more complicated tasks that ChatGPT can’t do.
  • It removes the necessity of constant googling and checking the documentation for libraries, which becomes an important benefit on more complex projects. Our team is following trends and has already implemented ChatGPT into the work. Using AI in software development requires technical expertise, so if you need help with your project or you don’t know how to use ChatGPT to write code, feel free to contact us. We will offer you a development team to meet your requirements or we will answer your questions.
  

  About author

  Alexander has been a part of the team since 2013 and is deeply interested in building top-notch web development products.

  Learn more about author