Trying to teach in the age of the AI homework machine

>I guess, I'm advocating for the Oxbridge/St. John's approach with under 10 class sizes where the proctor actually knows you and if you've done the work. And I know, that is not a cheap way to churn out degrees.

I could understand US tuition if that were the case. These days with overworked adjuncts make it McDonalds at Michelin star prices.

Over here in Finland, higher education is state funded, and the funding is allocated to universities mostly based on how many degrees they churn out yearly. Whether the grads actually find employment or know anything is irrelevant.

So, it's pretty hard for universities over here to maintain standards in this GenAI world, when the paying customer only cares about quantity, and not quality. I'm feeling bad for the students, not so much for foolish politicians.

After a short stint as a faculty member at a McU institution, I agree with much of this.

Provide machine problems and homework as exercises for students to learn, but assign a very low weight to these as part of an overall grade. Butt in seat assessments should be the majority of a course assessment for many courses.

>> (where everyone cheated anyways)

This is depressing. I'm late GenX, I didn't cheat in college (engineering, RPI), nor did my peers. Of course, there was very little writing of essays so that's probably why, not to mention all of our exams were in person paper-and-pencil (and this was 1986-1990, so no phones). Literally impossible to cheat. We did have study groups where people explained the homework to each other, which I guess could be called "cheating", but since we all shared, we tended to oust anyone who didn't bring anything to the table. Is cheating through college a common millenial / gen z thing?

I think this is where it's going to end up.

The masses get the cheap AI education. The elite get the expensive, small class, analog education. There won't be a middle class of education, as in the current system - too expensive for too little gain.

Cheap "universities" are fine for accreditation. Exams can be administered via in-person proctoring services, which test the bare minimum. The real test would be when students are hired, in the probationary period. While entry-level hires may be unreliable, and even in the best case not help the company much, this is already a problem (perhaps it can be solved by the government or some other outside organization paying the new hire instead of the company, although I haven't thought about it much).

Students can learn for free via online resources, forums, and LLM tutors (the less-trustworthy forums and LLMs should primarily be used to assist understanding the more-trustworthy online resources). EDIT: students can get hands-on-experience via an internship, possibly unpaid.

Real universities should continue to exist for their cutting-edge research and tutoring from very talented people, because that can't be commodified. At least until/if AI reaches expert competence (in not just knowledge but application), but then we don't need jobs either.

10 is a small number. There's a middle ground. When I studied, we had lectures for all students, and a similar amount of time in "work groups," as they were called. That resembled secondary education: one teacher, around 30 students, but those classes were mainly focused on applying the newly acquired knowledge, making exercises, asking questions, checking homework, etc. Later, I taught such classes for programming 101, and it was perfectly doable. Work group teachers were also responsible for reviewing their students' tests.

But that commercially oriented boards are ruining education, that's a given. That they would stoop to this level is a bit surprising.

Believe it or not, 300-person freshman lectures can be done well. They just need a talented instructor who's willing to put in the prep, and good TAs leading sections. And if the university fosters the right culture, the students mostly won't cheat.

But yeah, if the professor is clearly checked out and only interested in his research, and the students are being told that the only purpose of their education is to get a piece of paper to show to potential employers, you'll get a cynical death-spiral.

(I've been on both sides of this, though back when copy-pasting from Wikipedia was the way to cheat.)

Instructors and professors are required to be subject matter experts but many are not required to have a teaching certification or education-related degree.

So they know what students should be taught but I don't know that they necessarily know how any better than the administrators.

I've always found it weird that you need teaching certification to teach basic concepts to kindergartners but not to teach calculus to adults.

I totally agree. I think the neo-liberal university model is the real culprit. Where I live, Universities get money for each student who graduates. This is up to 100k euros for a new doctorate. This means that the University and its admin want as many students to graduate as possible. The (BA&MA) students also want to graduate in target time: if they do, they get a huge part of their student loans forgiven.

What has AI done? I teach a BA thesis seminar. Last year, when AI wasn't used as much, around 30% of the students failed to turn in their BA thesises. 30% drop-out rate was normal. This year: only 5% dropped out, while the amount of ChatGPT generated text has skyrocketed. I think there is a correlation: ChatGPT helps students write their thesises, so they're not as likely to drop out.

The University and the admins are probably very happy that so many students are graduating. But also, some colleagues are seeing an upside to this: if more graduate, the University gets more money, which means less cuts to teaching budgets, which means that the teachers can actually do their job and improve their courses, for those students who are actually there to learn. But personally, as a teacher, I'm at loss of what to do. Some thesises had hallucinated sources, some had AI slop blogs as sources, the texts are robotic and boring. But should I fail them, out of principle on what the ideal University should be? Nobody else seems to care. Or should I pass them, let them graduate, and reserve my energy to teach those who are motivated and are willing to engage?

The Open University in the UK started in 1969. Their staff have a reputation for good interaction with students, and I have seen very high quality teaching materials produced there. I believe they have always operated on the basis of remote teaching but on-site evaluation. The Open University sounds like an all-round success story and I'm surprised it isn't mentioned more in discussions of remote education.

Variations in this system are in active use in the US as well.

Do you feel it is effective?

It seems to me that there is a massive asymmetry in the war here: proctoring services have tiny incentives to catch cheaters. Cheaters have massive incentives to cheat.

I expect the system will only catch a small fraction of the cheating that occurs.

Can you tell us: Is "remote study" a relatively recent phenom in AU -- COVID era, or much older? I am curious to learn more. And, what is the history behind it? Was it created/supported because AU is so vast and many people a state might not live near the campus?

Also: I think your suggestion is excellent. We may see this happen in the US if AI cheating gets out of control (which it well).

Where I'm studying its proctored-online. They have a custom browser and take over your computer while you're doing the exam. Creepy AF but saves travelling 1,300 km to sit an exam.

Not even just large cities. Decent sized towns have them too, usually with local high school teachers or the like acting as proctors.

Proctoring services done well could be valuable, but it’s smaller rural and remote communities that would benefit most. Maybe these services could be offered by local schools, libraries, etc.

nope. too much impact on profit.

It is always interesting to me when people say they are "bad test takers". You mean you are bad at the part where we find out how much you know? Maybe you just don't know the material well enough.

caveat emptor - I am not ND so maybe this is a real concern for some, but in my experience the people who said this did not know the material. And the accommodations for tests are abused by rich kids more than they are utilized by those that need them.

I don't think I understand, as a terrible test taker myself.

The solution I use when teaching is to let evaluation primarily depend on some larger demonstration of knowledge. Most often it is CS classes (e.g. Machine Learning), so I don't really give much care for homeworks and tests and instead be project driven. I don't care if they use GPT or not. The learning happens by them doing things.

This is definitely harder in other courses. In my undergrad (physics) our professors frequently gave takehome exams. Open book, open notes, open anything but your friends and classmates. This did require trust, but it was usually pretty obvious when people worked together. They cared more about trying to evaluate and push us if we cared than if we cheated. They required multiple days worth of work and you can bet every student was coming to office hours (we had much more access during that time too). The trust and understanding that effort mattered actually resulted in very little cheating. We felt respected, there was a mutual understanding, and tbh, it created healthy competition among us.

Students cheat because they know they need the grade and that at the end of the day they won't won't actually be evaluated on what they learned, but rather on what arbitrary score they got. Fundamentally, this requires a restructuring, but that's been a long time coming. The cheating literally happens because we just treated Goodhart's Law as a feature instead of a bug. AI is forcing us to contend with metric hacking, it didn't create it.

IMO exams should be on the easier side and not require much computing (mainly knowledge, and not unnecessary memorization). They should be a baseline, not a challenge for students who understand the material.

Students are more accurately measured via long, take-home projects, which are complicated enough that they can’t be entirely done by AI.

Unless the class is something that requires quick thinking on the job, in which case there should be “exams” that are live simulations. Ultimately, a student’s GPA should reflect their competence in the career (or possible careers) they’re in college for.

> Many of them are neurodivergent who do miserably in exam conditions

Isn't this part of life? Learning to excel anyway?

> Many of them are neurodivergent who do miserably in exam conditions

I mean, for every neurodivergent person who does miserably in exam conditions you have one that does miserably in homework essays because of absence of clear time boundaries.

We have an Accessible Testing Center that will administer and proctor exams under very flexible conditions (more time, breaks, quiet/privacy, …) to help students with various forms of neurodivergence. They’re very good and offer a valuable service without placing any significant additional burden on the instructor. Seems to work well, but I don’t have first hand knowledge about how these forms of accommodations are viewed by the neurodivergent student community. They certainly don’t address the problem of allowing « explorer » students to demonstrate their abilities.

>Many of them are neurodivergent

if "many" are "divergent" then... are they really divergent? or are they the new typical?

I think having one huge exam at the end is the problem. An exam and assessment every week would be best.

Less stress at the end of the term, and the student can't leave everything to the last minute, they need to do a little work every week.

Professional proctored testing centers exist in many locations around the world now. It's not that complicated to have a couple people at the front, a method for physically screening test-takers, providing lockers for personal possessions, providing computers for test administration, and protocols for checking multiple points of identity for each test taker.

This hybrid model is vastly preferable to "true" remote test taking in which they try to do remote proctoring to the student's home using a camera and other tools.

A mix of feeling I’m “missing” some CS concepts and just general intellectual curiosity.

I am planning on doing a masters but I need some undergrad CS credits to be a qualified candidate. I don’t think I’m going to do the whole undergrad.

Overall my experience has been positive. I’ve really enjoyed Discrete Math and coming to understand how I’ve been using set theory without really understanding it for years. I’m really looking forward to my classes on assembly/computer architecture, operating systems, and networks. They did make me take CS 101-102 as prereqs which was a total waste of time and money, but I think those are the only two mandatory classes with no value to me.

> And why not a masters?

Not GP, but in my experience most MSC programs will require that you have substantial undergrad CS coursework in order to be accepted. There are a few programs designed for those without that background.

[dead]

> and in-person paper exams.

Yup. ~25 years ago competitions / NOI / leet_coding as they call it now were in a proctored room, computers with no internet access, just plain old borland c, a few problems and 3h of typing. All the uni exams were pen & paper. C++ OOP on paper was fun, but iirc the scoring was pretty lax (i.e. minor typos were usually ignored).

Thing is, this hits the scaling problem in education and fucking hard.

There’s such a shortfall of teachers globally, and the role is a public good, so it’s constantly underpaid.

And if you are good - why would you teach ? You’d get paid to just take advantage of your skills.

And now we have a tool that makes it impossible to know if you have taught anyone because they can pass your exams.

> I don't see why this is so hard, other than the usual intergenerational whining / a heaping pile of student entitlement.

You know that grading paper exams is a lot more hassle _for the teachers_?

Your overall point might or might not still stand. I'm just responding to your 'I don't see why this is so hard'. Show some imagination for why other people hold their positions.

(I'm sure there's lots of other factors that come into play that I am not thinking of here.)

I'm not too old either and in my university, CS was my major, we did group projects and in person paper exams as well.

We wrote c++ on paper for some questions and were graded on it. Ofcourse the tutors were lenient on the syntax they cared about the algorithm and the data structures not so much for the code. They did test syntax knowledge as well but more in code reasoning segments, i.e questions like what's the value of a after these two statements or after this loop is run.

We also had exams in the lab with computers disconnected from the internet. I don't remember the details of the grading but essentially the teaching team was in the room and pretty much scored us then and there.

> Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and and extending it to more courses in the fall.

Excellent approach. It requires a big buy-in from the school.

Thanks for suggesting it.

I'm doing something for one kind of assignment inspired by the game "bashcrawl" where you have to learn Linux commands through an adventure-style game. I'm bundling it in a container and letting you submit your progress via curl commands, so that you pass after having run a certain set of commands. Trying to make the levels unskippable by using tarballs. Essentially, if you can break the game instead of beating it honestly, you get a passing grade, too.

>Our school converted a computer lab to a programming lab. Computers in the lab have editors/compilers/interpreters, and whitelist documentation, plus an internal server for grading and submission. No internet access otherwise. We've used it for one course so far with good results, and extending it to more courses in the fall.

As an higher education (university) IT admin who is responsible for the CS program's computer labs and is also enrolled in this CS program, I would love to hear more about this setup, please & thank you. As recently as last semester, CS professors have been doing pen'n paper exams and group projects. This setup sounds great!

Isn't auto-grading cheating by the instructors? Isn't part of their job providing their expert feedback by actually reading the code the students have generating and providing feedback and suggestions for improvement even at for exams? A good educational program treats exams as learning opportunities, not just evaluations.

So if the professors can cheat and they're happy about having to do less teaching work, thereby giving the students a lower-quality educational experience, why shouldn't the students just get an LLM to write code that passes the auto-grader's checks? Then everyone's happy - the administration is getting the tuition, the professors don't have to grade or give feedback individually, and the students can finish their assignments in half an hour instead of having to stay up all night. Win win win!

Is it in a Faraday cage too or do you just confiscate their phones. Or do you naively believe they aren't just using AI on their phones?

> I know that I wouldn't have wanted to type a large pile of code into a web anything.

I might not have liked that, but I sure would have liked to see my useless classmates being forced to learn without cheating.

You can provide vscode, vim and emacs all in some web interface, and those are plenty good enough for those use cases. Choosing the plugin list for each would also be a good bikeshedding exercise for the department.

Even IntelliJ has gateway

Yeah, I've had teachers like that, who tell you that you're a "waste of life" and "what are you doing here?" and "you're dumb", so motivational.

I guess this "tough love" attitude helps for some people? But I think mostly it's just that people think it works for _other_ people, but rarely people think that this works when applied to themselves.

Like, imagine the school administration walking up to this teacher and saying "hey dum dum, you're failing too many students and the time you've spent teaching them is a waste of life."

Many teachers seem to think that students go to school/university because they're genuinely interested in motivated. But more often then not, they're there because of societal pressure, because they know they need a degree to have any kind of decent living standard, and because their parents told them to. Yeah you can call them names, call them lazy or whatever, but that's kinda like pointing at poor people and saying they should invest more.

And don't forget to get on their case with accusations of technology use that equate to the Turing test

Grandpa can help with that too

These conversations are always eye-opening for the number of people who don’t understand homework. You’re exactly right that it’s practice. The test is the test (obviously) and the homework is practice with a feedback loop (the grade).

Giving people credit for homework helps because it gives students a chance to earn points outside of high pressure test times and it also encourages people to do the homework. A lot of people need the latter.

My friends who teach university classes have experimented with grading structures where homework is optional and only exam scores count. Inevitably, a lot of the class fails the exams because they didn’t do any practice on their own. They come begging for opportunities to make it up. So then they circle back to making the homework required and graded as a way to get the students to practice.

ChatGPT short circuits this once again. Students ChatGPT their homework then fail the first exam. This time there is little to do, other than let those students learn the consequences of their actions.

Completely agree - I always thought the framing of "exercises" is the right one, the point is that your brain grows by doing. It's been possible for a long time to e.g. google a similar algebra problem and find a very relevant math stackexchange post, doesn't mean the exercises were useless.

"The fact that forklift truck can lift over 500kg should tell you how worthwhile it is for me to go to a gym and lift 100kg." - complete non-sequitur.

> A lot of homework can be easily done by AI

Then maybe the homework assignment has been poorly chosen. I like how the article's author has decided to focus on the process and not the product and I think that's probably a good move.

I remember one of my kids' math teachers talked about wanting to switch to in inverted classroom. The kids would be asked to read a some part of their textbook as homework and then they would work through exercise sheets in class. To me, that seemed like a better way to teach math.

> But things have changed now.

Yep. Students are using AIs to do their homework and teachers are using AIs to grade.

Yep, making time to sit down to do homework, forming an understanding of planning the doing part, forming good habits of doing them, knowing how to look up stuff, in a book index or on Wikipedia or by searching or asking AI. The expectation is still that some kind of text output needs to be found and then read, digested.

Let me rephrase their point, then:

The fact that AI can replace the work that you are measured on should tell you something about the measurement itself.

The goal of learning should be to enrich the learner. Instead, the goal of learning is to pass measure. Success has been quietly replaced with victory. Now LLMs are here to call that bluff.

Its a problem of incentives. For many courses the psets make up a large chunk of your grade. Grades determine your suitability for graduate school, internships, jobs, etc. So if your final goal is one of those then you are highly incentivized to get high grades, not necessarily to learn the material.

> I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.

I see both sides of this. When I was a teenager, I went to a pretty bad middle school where there were fights everyday, and I wasn’t learning anything from the easy homework. On the upside, I had tons of free time to teach myself how to make websites and get into all kinds of trouble botting my favorite online games.

My learning always hit a wall though because I wasn’t able to learn programming on my own. I eventually asked my parents to send me to a school that had a lot more structure (and a lot more homework), and then I properly learned math and logic and programming from first principles. The upside: I could code. The downside: there was no free time to apply this knowledge to anything fun

>I'm 62, and what allowed me to follow my curiosity as a kid was that the school lessons were finite, and easy enough that I could finish them early, leaving me time to do things like play music, read, and learn electronics.

Yeah I feel like teachers are going to try and use LLMs as an excuse to push more of the burden of schooling to their pupils homelife somehow. Like, increasing homework burdens to compensate.

Spare time, haha, most people nowadays have a hard time having some dead time. The habitual checking of socials or feeds has killed the mind wandering time. People feel uncomfortable or consiser life boring with the device induced dopamine fix. Corporations got us by the balls.

Still better than the typical classroom experience. And you can always ask again, there's no need to avoid offending the person who has a lot of power over you.

Wasn't this the promise of MOOCs in the 2010s?

Yes, I think curiosity dies well before university for most students.

The specific examples I recall most vividly were from 4th grade and 7th grade.

Indeed. I never really used AI until recently but now I use it sometimes as a smarter search engine that can give me abstracts.

Eg. it's easy to ask copilot: can you give me a list of free, open source mqtt brokers and give me some statistics in the form of a table

And copilot (or any other ai) does this quite nicely. This is not something that you can ask a traditional search engine.

Offcourse you do need to know enough of the underlying material and double check what output you get for when the AI is hallucinating.

Do you have capacity for more developers? I’ve been wanting to help make this for a long time

Back when I was doing my BSc in Software Engineering, we had a teacher who did her Data Structure and Algorithms exams with pen and paper. On one of them, she basically wrote 4 coding problems (which would be solved in 4 short ~30 LOC).

We had to write the answer with pen and paper, writing the whole program in C. And the teacher would score it by transcribing the verbatim text in her computer, and if it had one single error (missed semicolon) or didn't compile for some reason, the whole thing was considered wrong (each question was 25% of the exam score)

I remember I got 1 wrong (missed semicolon :( ) and got a 75% (1-100 pointing system). It's crazy how we were able to do that sort of thing in the old days.

We definitely exercised our attention to detail and concentration muscles with that teacher.

Yes, pen and paper. The approach is to pseudocode the solution, minor syntax errors aren’t punished (and indeed are generally expected anyway). The point is to simply show that you understand and can work through the concepts involved, it’s not being literally compiled.

Writing a small algorithm with pen & paper on programming exams in universities of all sizes was still common when I was in uni in the 2010s and there’s no reason to drop that practice now.

Yes, pen and paper.

> rather than giving them the flexibility they deserve as grown adults

I have had so many very frustrating conversations with full grown adults in charge of teaching CS. I have no faith at all that students would be able to choose an appropriate method of study.

My issue with the instruction is the very narrow belief in the importance of certain measurable skills. VERY narrow. I won’t go into details, for my own sanity.

> All of the best professors I had either did not grade homework or weighted it very small and often on a did-you-do-it-at-all basis and did not grade attendance at all. They provided lectures and assignments as a means to learn the material and then graded you based on your performance in proctored exams taken either in class or at the university testing center.

I have the opposite experience - the best professors focused on homework and projects and exams were minimal to non-existent. People learn different ways, though, so you might function better having the threat/challenge of an exam, whereas I hated having to put everything together for an hour of stress and anxiety. Exams are artificial and unlike the real world - the point is to solve problems, not to solve problems in weirdly constrained situations.

I don’t disagree, but in most cases degrees are handed out based on grades which in turn are based on homework.

I agree that something will have to change to avert the current trend.

Grading homework has two reasonable objectives:

Provide an incentive for students to do the thing they should be doing anyway.

Give an opportunity to provide feedback on the assignment.

It is totally useless as an evaluation mechanic, because of course the students that want to can just cheat. It’s usually pretty small, right? IIRC when I did tutoring we only gave like 10-20% for the aggregate homework grade.

How do you suggest we measure whether the students have actually learned the stuff then?

> the faculty felt that Java IDEs with autocomplete were doing a disservice to learning

Sounds laughably naive now, doesn’t it?

If you’re hiring humans just to use AI, why even hire humans? Either AI will replace them or employers will realize that they prefer employees who can think. In either case, being a human who specializes in regurgitating AI output seems like a dead end.

Even if you just use AI, you need to know the right prompts to ask.

Would you rather be the guy using AI as a crutch or the guy who actually knows how to do things without it?

To be clear, I think that most college degrees were generally low value (even my own), but still had some value. The current trend will be towards zero value unless something changes.

It doesn't matter if your boss's policy is to require a degree.

Confession. I became disillusioned with a teacher of a subject in school, who I was certain had taken a disliking to me.

I tested it by getting hold of a paper which had received an A from another school on the same subject, copying it verbatim and submitting it for my assignment. I received a low grade.

Despite confirming what I suspected, it somehow still wasn't a good feeling.

Is there even such a thing as the "science of business"? One can form a hypothesis, and then conduct an experiment, but the experimental landscape is so messy that eliminating all other considerations is impossibly hard.

For example, there's a popular theory that the single major factor in startup success is timing - that the market is "ready" for ideas at specific times, and getting that timing right is the key factor in success. But it's impossible to predict when the market timing is right, you only find out in retrospect. How would you ever test this theory? There are so many other factors, half of which are outside the control of the experimenter, that you would have to conduct the experiment hundreds of time (effectively starting and failing at hundreds of startups) to exclude the confounding factors.

To be honest, I found "the material" irrelevant, mostly. There's vast swathes of papers written about obscure and tiny parts of the overall subject. Any given paper is probably correct, but covering such a tiny part of the subject that spending the time reading all of them is inefficient (if not impossible).

Also, given that the subject in question is "business", and the practice of business was being changed (as it is again now) by the application of new technology, so a lot of what I was reading was only borderline applicable any more.

MBAs are weird. To qualify to do one you need to have years of practical experience managing in actual business. But then all of that knowledge and experience is disregarded, and you're expected to defer to papers written by people who have only ever worked in academia and have no practical experience of what they're studying. I know this is the scientific process, and I respect that. But how applicable is the scientific process to management? Is there even a "science" of management?

So, like all my colleagues, I jumped through the hoops set in front of me as efficiently as possible in order to get the qualification.

I'm not saying it was worthless. I did learn a lot. The class discussions, hearing about other people's experiences, talking about specific problems and situations, this was all good solid learning. But the essays were not.

Interesting question. It seems to me that the entire business academia could be following the method I've outlined and no-one would notice. Or care.

It's not like the hard sciences - no-one is able to refute anything, because you can't conduct experiments. You can always find some evidence for any given hypothesis, as the endless stream of self-help (and often contradictory) business books show.

None of the academics I was reading had actually run a business or had any practical experience of business. They were all lifelong academics who were writing about it from an academic perspective, referencing other academics.

Business is not short of actual independent thought. Verification is the thing it's missing. How does anyone know that the brilliant idea they just had is actually brilliant? The only way is to go and build a business around it and see if it works. Academics don't do that. How is this science then?

No, we already have autonomous cars driving around even though they've already killed people.

The funny thing is my first thought was "maybe reduced nominal productivity by increased throughness is exactly what we need when evaluating potential tumors". Keeping doctors off autopilot and not so focused that radiologists fail to see hidden gorillas in x-rays. And yes that was a real study.

Have you tried image generation? It can easily apply high level concepts from one area to another area and produce something that hasn't been done before.

Unless you loosen the meaning of statistical average so much that it ends up including human creativity. At the end of the day it's basically the same process of applying an idea from one field to another.

Most humans are not Da Vinci, Einstein, Kant, etc. Does that make them not valuable as humans?