Comparing Human and Artificial Intelligence Approaches to Providing Writing Assistance

Project Title

The Future of Writing Centers: Comparing Human and Automated Approaches to Providing Writing Assistance

Faculty Mentor(s)

Project Description

Will artificial intelligence replace or augment human labor?

My research compares human and artificial intelligence (AI) approaches to providing writing assistance for college student essays. The results of the project will inform ethical AI design choices, so that future technology does not negatively impact the learning process of new academic writers, and the job security and integrity of intellectual workers.

In 2022, OpenAI’s ChatGPT, powered by the Large Language Model (LLM) GPT-3, revolutionized the way in which human interact with advanced computer systems. Taking in natural language prompts from the user, ChatGPT can output surprisingly human-like responses. Since then, GPT-3 and similar technologies have been finetuned to provide a wide range of services, from providing basic customer services, to autogenerating nuanced artworks and code – all of which has significantly impacted different markets for human labour.

Our study specifically examines the field of providing writing assistance to college-level writings. We collect and compare writing evaluations for 60 college student-written papers across the disciplines from human experts (Writing Associates), and ChatGPT. The results of this study will further inform ethical LLM design choices, as well as the future of work at Writing Centers.

How long have you been conducting research?

My mentors and I hatched the idea in December 2022, started preparing for the project late April 2023, and officially began conducting research mid-June 2023 post IRB exemption.

Why is your research important?

In order to supply training data for LLMs, large private corporations regularly scrape the web for publicly available information from users. As a result, LLM engines, which are in essence capital goods, become more and more sophisticated and capable of providing a basic range of human services. Yet, most of the general population does not accurately understand the mechanism of such engines or how they may impact their future of work.

What does the process of doing your research look like?

On a typical day, I recruit participants, design LLM prompt templates, as well as read new literature. I also regularly meet up with my mentors to talk about the progress of the research and discuss new findings.

What knowledge has your research contributed to your field?

My research is still in its data collection process, and thus we have not made any new discoveries. We expect to start data analysis mid-Fall semester.

In what ways have you showcased your research thus far?

I presented the current state of my research at the 2023 Oberlin Summer Research Symposium.

How did you get involved in research? What drove you to seek out research experiences in college?

Sometime around the end of my freshman year, after having taken a bunch of courses in computer science and politics, I decided that I wanted to major in both fields. After talking to my professors, I became interested in answering social science questions about technology, and found that doing research is the best way to do so!

What is your favorite aspect of the research process?

My favorite aspect of the project is conducting the literature survey. Over the summer, I learned a lot about the social science side of computer science, and where my different disciplines of interest can meaningfully intersect.

How has working with your mentor impacted the development of your research project? How has it impacted you as a researcher?

My mentors have taught me a lot about how to manage the logistics of research: writing protocols, reaching out to and keeping track of participants, and ethical research practices. From doing research and talking to my mentors regularly, I’ve gained a lot of confidence about going into academia post-grad!

How has the research you’ve conducted contributed to your professional or academic development?

Through this research project, I was able to identify specific discourses that I am interested in contributing to, as well as directions that I could take with future projects.

What advice would you give to a younger student wanting to get involved in research in your field?

Before I was able conduct my current research project, I’ve proposed a bunch of other that didn’t make it out of the drawing board. In moments of self-doubt, my friends and professors have been a great source of support, both in helping me refine my approach, as well as building up my confidence. Therefore, I highly recommend communicating your interests and goals to the community of people around you – people at Oberlin are always ready to help.