OpenAI, ChatGPT, & What's Coming With AI
Jan 19, 2023 • 6 Minute Read • Elizabeth Spranzani, Chief Technology Officer
See All Our Insights
Github Copilot: An AI Time-Saving Superpower
Jun 06, 2023 • 4 Minute Read • Adrian Steinsleger, Associate Architect, Front End
Picture having a programming assistant that can learn and use your coding style while following best practices. Github Copilot is a generative AI tool that helps software developers improve productivity and efficiency at work.
Our team conducted a survey of 67 developers with varying levels of experience who used Copilot for a month. The goal was to determine if the tool lives up to the hype and can truly help us code faster and improve our skills. In this article, we'll provide an overview of Copilot and share our survey results.
Copilot At-a-Glance
Created by GitHub and OpenAI, Copilot focuses mainly on providing suggestions for lines of code or complete functions directly inside integrated development environments (IDEs). It's based on Codex, a large language model (LLM) from OpenAI, which is trained on natural language text and a vast data set of public code from GitHub and other sources. Copilot has upcoming features, including context-aware conversations, similar to other tools like ChatGPT. However, Copilot's primary focus is on code completion.
Github Copilot is trained using programming languages found in public code repositories. The quality of suggestions a user receives may be influenced by the amount and variety of training data available for each language.
As an example, JavaScript is one of GitHub Copilot's top-supported languages and is widely used in public repositories. Languages that are underrepresented in public repositories could generate fewer or weaker suggestions, however, it also makes suggestions based on a developer's previously written code. As a developer uses it more, the tool becomes more intelligent and even starts to follow their personal coding preferences in addition to standard ones.
This tool is dynamic as it can provide suggestions in real time by predicting what the developer is attempting to achieve and suggesting possible solutions after a code comment and the descriptive goal is entered.
Also, available for VS Code, there is an extension named Copilot Labs, which consists of a sidebar companion for Copilot, and features four panels with extended tools like code language translation, explaining highlighted code, generating test units, or creating documentation.
Our Experience
After using Copilot for a month, we ran a survey among our developers to evaluate their self-perception of improved productivity, efficiency, and satisfaction.
Although we are still comparing these results to data-driven metrics, they were compelling enough to convince us to incorporate Copilot into our team's standard tool set. Some of our team members had already been using the closed-pre-release version of Copilot when it was initially launched on November 22nd.
Here are some highlights from the survey answered by a pool of 67 developers with experience varying between less than five, five to 10, and 10 to 15 years of programming:
- Copilot is generally well-received, with many respondents reporting positive experiences. It appears to be particularly helpful for boilerplate code, repetitive tasks, and code generation.
- Copilot's usefulness varies depending on the user's role and experience level. Developers with more experience seem to find it beneficial for improving their code quality and efficiency.
- The tool seems to be popular among backend, frontend, and full-stack developers, indicating its versatility across different development domains.
- Copilot appears to be more effective for users working in remote and hybrid work environments, potentially due to the increased reliance on self-guided learning and problem-solving in these settings.
- Despite many positive experiences, some users didn't find Copilot useful or didn't have enough opportunity to evaluate its effectiveness, due to role-specific events during the evaluation period, or to limitations to IDE versions. This suggests that the tool's usefulness may vary depending on individual preferences, coding habits, or project requirements.
- For users with less than five years of experience, 58% found that Copilot helped them save around one hour per week. This suggests that Copilot is particularly beneficial for less experienced developers in saving time during the development process.
- Of the users with five to 10 years of experience, 47% reported saving around one hour. However, 29% experienced even greater time savings of two hours or more. This indicates that Copilot can also be helpful in saving time for developers with an intermediate level of experience.
- For users with 10-15 years of experience, the distribution of time savings is more diverse - 47% saved less than one hour, 33% saved around one hour, and 20% saved two hours or more. This suggests that the time-saving benefits of Copilot for developers with this level of experience may vary. It may indicate the assignments this group usually has, combined with their seniority, makes the usage of Copilot less frequent or relevant.
- Users with 15+ years of experience show a more balanced distribution of time savings, with 38% saving one hour or less, while 25% saved two hours or more. This indicates that developers with extensive experience have varying opinions on the time-saving benefits of Copilot.
Summary
The insights show that Copilot is a time saver across all experience levels, with the most significant benefits reported by users with less than five years of experience and five to 10 years of experience. Copilot's impact on time savings seems to vary more for users with 10-15 years of experience and 15+ years of experience.
Initially, saving only one hour per week may not appear substantial, however, in a company with 50 developers working 50 weeks a year, the total impact would be 2,500 hours. Additionally, considering the previously mentioned upcoming features and the tool's potential for growth, we can anticipate that its positive impact and cost savings will continue to improve over time.
Also, after using Copilot, developers find it challenging to return to programming without its assistance, primarily due to its cost-benefit ratio.
As generative AI tools become more widely available, it's fundamental to learn how we can use them to our advantage. The future is full of exciting possibilities, including new products and capabilities, and we look forward to seeing what comes next.