Turning ChatGPT Into Its Own Critic – Cardinal Nation comes clean on our last article

Yesterday, you may have noticed the story Cardinal Nation ran detailing how the quickly-rising ChatGPT artificial intelligence technology is impacting education. The article – entitled “ChatGPT: The New Challenge Facing Education At MHS” – discussed how ChatGPT can be used by students in dishonest ways, citing the opinions of nameless administrators, teachers, and students.

What you may not have realized is that the entire article was actually generated by the very same AI technology which it was criticizing: ChatGPT.

We’ll show you how it was done, and also share the opinions of (actual) staff and administration about this new technology.

ChatGPT – How does it work?

The process of using ChatGPT is surprisingly easy, and creating the article only took a matter of minutes. Navigating to the website of OpenAI, the company behind ChatGPT, we were able to create an account. While OpenAI has made access to ChatGPT free for now, there are indications that this could soon change.

After creating an account, it was simply a matter of crafting a prompt to provide ChatGPT that would produce the article we wanted. For the article, we provided this prompt:

Using a journalistic tone, write a lengthy article for a school newspaper. The topic of the article is the negative effects of ChatGPT on high school education.

In the article, explain what ChatGPT is. Then, include some quotes from a nameless school administrator and also a student. Finally, discuss how the school has tried to stop cheating that has resulted because of ChatGPT. The name of the high school is Mentor High School.

While the program can work with relatively little text, providing it with more information can produce results closer to what is wanted. For instance, the prompt we used contained some context about where it would be published, a general outline of how the article should be structured, and also the name of Mentor High. Because of these details, we got a far more informative and convincing article than if we had simply given it a generic prompt.

After pressing the “Enter” key, it only took a matter of seconds for the entire article to be produced. While ChatGPT does have a “regenerate” button, which will create a new response, the article we published was exactly what it produced during the first run. While its response did not come with a title, creating one was as simple as asking it to “Generate a title”.

The field of generative AI is not limited to text, and we utilized another product from OpenAI to create some artwork for the article. Known as DALL·E 2, this AI program can take a prompt and generate images, ranging from ultra-realistic 3D renders to “hand drawn” pencil sketches. Crafting an image using this program was far more difficult than using ChatGPT, however. After adjusting the wording a few times, we finally got an image that suited the article.

Mentor High School educators react

While ChatGPT is impressive in capability and usability, what do Mentor teachers and administration think about this technology?

“I suspect students are using it,” says Mrs. Amy Roediger, a chemistry teacher and instructional coach at MHS, “but I haven’t really been looking for it in the work of my students.”

“I have not noticed students using Chat GPT,” agrees Mrs. Tracy Coleman, MHS English Department chair and teacher. “Right now [AP Seminar] students are writing their Performance Task One papers that will be scored by the College Board and will be part of students’ AP scores. As a result, the stakes are high for them with these papers…We are also talking about how ChatGPT will present facts, but it will not provide the kinds of commentary students need to offer where they put perspectives into conversation with one another. As a result, a paper generated by Chat GPT would score low on the AP Seminar rubric.”

Mrs. Coleman mentioned that within the context of writing assignments, ChatGPT has some major limitations.

“Chat GPT can write an essay for a student,” Mrs. Coleman says, “but the essay has no ‘voice.’ The work is entirely based on facts available on the internet; there is no heart to it. ChatGPT can solve a math problem but it does not always do so correctly.”

Similarly, Mrs. Roediger shared a twitter thread of ChatGPT being challenged by a simple math problem, showing another limitation.

Of course, this is not the first time new technology has had an impact on educational integrity among MHS students, given the rise of PhotoMath, Sparknotes, and other resources.

“Using ChatGPT to represent ideas as one’s own when they are not would be similar to using PhotoMath to do math homework,” Mrs. Roediger says. “Whether or not the impact on integrity will be the same remains to be seen.”

“I do think that ChatGPT will be of the same magnitude,” Mrs. Coleman says. “I am not, however, taking a ‘the sky is falling’ approach to ChatGPT. To your point about there being other resources students can use to cheat, this is not a new issue in education. This is just the most recent incarnation of how, as technology evolves, educators need to be sure we are aware of what is available to students and creating assignments that require students to demonstrate their own thinking.”

Alongside their concerns regarding academic integrity, MHS teachers and Mentor administration also believe ChatGPT has potential to be used in beneficial ways within the classroom. Mentor Public Schools superintendent Mr. Heath, who responded to Cardinal Nation’s questions together with MHS principal Mr. Jason Crowe and Mentor Schools Director of Informational Technology and Innovation Mr. Mike Lynch issued an exclusive joint statement to Cardinal Nation:

“We believe ChatGPT does have a place in our classrooms. We will be working on how to best integrate ChatGPT and other Artificial Intelligence programs into our curriculum as we learn more.”

“As long as students use it for the positives,” Mrs. Coleman says, “ChatGPT could be a helpful tool. As an example, using Sparknotes, Litcharts, or other online resources can be helpful when students use them in conjunction with actually reading a text. When used appropriately, ChatGPT could be helpful to students; it just should not be used to bypass the writing process.”

Mrs. Roediger believes that “this might be another ‘time will tell’ situation” and that “there are many, many resources available to find explanations for difficult concepts…I would turn to a lot of other sources for help before ChatGPT.”

While focus has been on students using ChatGPT, the work of teachers and other employees also has potential to be disrupted. Nevertheless, Superintendent Heath sees potential in the technology.

“Using technology properly allows us to manage our work in a variety of ways…” Superintendent Heath tells Cardinal Nation. “Our hope is that these new technologies help our teachers with their work loads on a daily basis.

We are excited to see how this technology evolves and how we can use it in positive ways in the future”.

However, teachers at MHS remain more skeptical about its use cases.

“I have seen some lesson plans generated by ChatGPT and they were mediocre at best,” Mrs. Roediger says. “Again, other better resources are available to generate great ideas. Except in the case of objective measures like multiple choice questions with one right answer, I also wouldn’t use a tool like this to evaluate student work.”

“I would not use ChatGPT to grade papers because grading students’ writing allows me a window into their strengths and opportunities for improvement so I can design instruction to help them improve.” Mrs. Coleman agrees. “I am open to using any new technology that can be helpful for ideas, but I strongly believe in creating assignments for my own students’ needs rather than looking for generic approaches to topics.”

The skepticism among teachers is further evidence that the AI still needs to be developed further before it can be considered a serious assistant technology.

So…what do we do with ChatGPT going forward?

With the potential impact ChatGPT may have on student integrity, what can actually be done about it?

Globally, companies and individuals have been building tools to identify if a piece of text was generated by ChatGPT. For example, Princeton University student Edward Tian, whose story was reported in NPR’s Planet Money, developed a tool called GPTZero.me to detect AI generated text. This tool looks at trends in vocabulary, sentence structure, and length to determine if the article was written by man or machine, which may be useful for teachers wishing to deter ChatGPT usage.

After using this tool on the article we ran yesterday, it did indicate that AI was likely used in the creation of the article, showing its potential usage. However, after making some slight changes to the diction and format of the generated article, GPTZero.me was less confident in its results. From this, it is clear that both ChatGPT and its deterrent both have room for improvement.

Mentor Schools itself has also been taking some action to limit the negative academic impacts of ChatGPT. Mrs. Coleman and Mrs. Roediger both agree that immediate curriculum changes are not warranted.

“We cannot expect that students will not explore using ChatGPT,” Mrs. Coleman says, “so we need to work as a staff/district to help students understand appropriate use and also know that ChatGPT has pitfalls. [We] should be talking about the types of assignments that will not promote the use of ChatGPT, such as writing assignments rather than recall questions.”

That being said, teachers are still taking some actions.

“Next week,” Mrs. Coleman says, “I (along with some other staff members) am attending a webinar offered by Turnitin on academic integrity in light of developments in AI. Companies and teachers are already discussing ways to promote academic honesty in a world with Chat GPT.” 

“Our IT department is also sharing specific information about ChatGPT with the entire staff in their next monthly newsletter,” Mr. Heath tells us of actions at the district level. “Members of our high school English Department and our Instructional Coaches are attending webinars to learn the best ways to incorporate ChatGPT in our classrooms.”

According to Mr. Heath, this is also on the agenda for the next Superintendent Roundtable at the Educational Service Center of Northeast Ohio.

As for official district policy, it still remains unclear how exactly it will adapt to this.

“We already have board policies in place dealing with student integrity and plagiarism (Policy 6.19 – Student Code of Conduct),” Mr. Heath reminds us. “We will take a close look at the NEOLA policies that we will be transitioning to for the 2023-24 school year to make sure the student integrity language is aligned to our practices in Mentor Schools.”

With the language models that are at the core of this technology growing at an exponential rate, future improvements in AI text-generation are nearly guaranteed. However, what remains less certain is how we as a school, a district, or even as a society, will utilize and adapt to this disruptive new technology. It has the capability to creatively empower people, and make drastic improvements to the efficiency of their workflow.

I even occasionally used ChatGPT when writing this article to generate ideas and get feedback. The cover image for this article, similar to the last one, was also generated by AI. Yet, this technology also has the potential to severely impede the writing abilities of students across the nation and compromise academic integrity.

Regardless of who we asked, the one point everyone agreed on was that the future of education will certainly be made more interesting and complicated by this emergent technology.