Who remembers Eliza?
Eliza was a program you could type a query into and it would attempt to have a conversation with you. It dated to the 1960s and there were implementations on most of the 1970s personal computers, including the Tandy TRS-80 line. It was quite amusing but trivially evident you were talking at a program, not an intelligent thing.
Wolfram Alpha is another example and one that's still around. It can and does produce fairly complex answers and, when the question is related to a mathematics or physics-related question is quite useful.
Today's "latest round" of this is ChatGPT and schools are allegedly "banning it", blocking access and similar.
Let's differentiate.
Wolfram Alpha is nearly always correct because mathematics, chemistry and physics are deterministic. That is 2H2 + O2 -> 2H2O + e (energy) always. It is never different and never will be. If you ask it how to calculate the surface of a sphere it will give you the formula -- and it will be correct, provided it parses the request correctly, because there is only one answer and that answer is always the same.
But you can easily differentiate that you're not talking to a chemistry professor; ask it something "out of scope" for chemistry, physics or mathematics and you'll see immediately what I mean. The chemistry professor is going to either tell you they don't know or will try to extrapolate given their experience which might include the fact that some jackass almost ran them over with their car this morning on the way to work, that their dog eat a bag of food that had poison in it from China or similar. Wolfram will never do that because it doesn't know how to go "out of scope."
Thus you'll never mistake one for the other, and Wolfram doesn't try.
So how is it that ChatGPT is worth blocking in educational environments?
The problem lies not with the program; it lies with the instructors.
If you cannot, as an instructor, design your testing and work materials such that you can detect that which doesn't require "out of scope" thought in the classroom and during examinations you're not very good at what you do and further, you're worthless compared against a computer when it comes to actually teaching.
Who's seen the first of the "Kelvin Series" of Star Trek movies? Young Spock is being tested by a computer; it recites questions and he provides answers. A similar testing process is displayed on Vulcan during The Voyage Home after McCoy returns Spock's katra and he has to be retrained. You'll note that with the exception of the last question in the latter -- "How do you feel?" -- every query and response is a physical fact. Yes, even the philosophy question is fact; it is from a text, it was learned, and thus is fact in terms of what was learned -- if you might disagree with it.
So-called "artificial intelligence" does not exist; it is a marketing term used by those who think they can replace someone who is in fact an NPC; that is, a programmed automaton. It is utterly true that a computer can do that and if that's what you are in any particular domain of your existence then you are an NPC -- not a thinking being.
Actual intelligence -- of any level -- is demonstrated only by "out of scope" results to the question posed. If whatever returns the results does not exceed the scope of factual knowledge it starts with them it is not thinking. It is nothing more than a pattern-matching device -- perhaps a very fast one (and computers are very fast) but it is displaying nothing more than the ability to do pattern matching at high speed.
ChatGPT and the rest of these things, in short, cannot even display the thinking power of my cat. Said cat has figured out how to open a drawer; a skill that is most-assuredly "out of scope" with how a cat historically and mechanically thinks. A cat instinctively knows how to eat, drink, groom itself and find a suitable place to relieve itself (rather than peeing all over the floor) and unlike a dog you don't have to teach it that filth is bad; it knows it is bad and deliberately acts to avoid being filthy.
The cat, sensing either by noise or smell that something is beyond a point that appears to be absolutely blocked from access will attempt to resolve that problem. Dogs will display this as well. Both may get some of that from observation of other creatures (e.g. you) opening said door or cabinet but they are curious enough and have enough out-of-scope processing capacity to attempt to solve said problem even though nothing in their natural environment has given them a pattern to match in that regard, nor have you consciously taught it to them (and thus directly instructed them in said pattern.)
That is out-of-scope processing and a number of animals have a limited capacity to do it. Crows, for example, have demonstrated a remarkable ability to achieve this including the use of tools and even more-astounding they can and do assemble a tool out of parts. You do the same thing; you have a wrench handle and socket, and you need to remove a bolt. You assemble the required tool from the parts and then use it. You think nothing of this but that synthesis is out-of-scope thought which is the definition of actual intelligence.
Likewise it is known that a crow will figure out that a stick bent a certain way or combined with some other element it can put together will extract bugs from a hole, which the crow desires to eat. The cognitive leap here is obvious to us but utterly beyond the capacity of something like ChatGPT, even when prodded.
That is a Turing test; that is, how can you determine whether or not whatever you're interacting with actually thinks. If that thing does not then it is a NPC in "game parlance" such as role-playing games (e.g. Dungeons and Dragons, etc.) Said things may look like intelligent beings but are not in that they have a base of information and either refuse to engage or are incapable of synthesis beyond that.
Recently I "engaged" ChatGPT in a conversation about a rather topical thing. It spewed pablum, obviously programmed to say that on the topic, back at me. Said information has been scientifically falsified although there is some question as to exactly how seriously-so and how deterministically-so.
Nonetheless what it said in response was false as it made a statement of fact, not nuance or qualified belief.
I challenged it directly including the contrary facts, which are facts as they came from a public medical study published by an extremely well-respected source: The Cleveland Clinic.
In response it apologized for its previous response not being accurate and then went on to repeat the false claim while deliberately evading the contrary evidence in its reply.
In other words it has been instructed how to lie by refusing to synthesize into its response the new information -- and its not very good at it either. Whether that's because its incapable of synthesis or has chosen not to is immaterial to the outcome which is that it presented a second false set of statements as the point was deliberately evaded.
When it lied back to me I repeated the study results focusing on the specifics in an attempt to prevent it from evading the answer. It again apologized and admitted my statement was true, in other words it admitted it lied without actually using those words, and rather than stop there it then tried to obfuscate the truth once again by including irrelevant possibilities which, while they might be true, were not part of the discussion!
I again nailed it and this time stated that "repeating something several times does not make it more true, or make a false statement true." I then included other examples which were true and related (e.g. measles) and accused it directly of being a programmed response rather than dialogue (in other words, I directly challenged it as a robot regurgitating someone else's opinion rather than processing information and coming to a deduction.) I then included that if in fact its statements were true they possibly constituted evidence that its claims would support that path of action being actively harmful in certain circumstances (e.g. health care workers.)
It apologized again for its statements possibly not being accurate or helpful (notice the addition.) It also once again dodged the actual question and completely ignored the additional risk it promoted itself.
At that point I gave up.
If you recognize this pattern among people over the last three years you're not alone. But the conversation I had was with a machine folks -- and it is incapable of deductive reasoning because it is a machine. It will never exhibit even the capacity of my cat to open a drawer because said cat thinks there might be something inside worth chasing, playing with or eating. Said cat, once it recognizes that the drawer is empty, either (1) climbs in because its intent was to sit inside in the first place (cats like boxes, you know) or (2) once it realizes there is nothing in there to eat or play with it dismisses the activity and walks away.
The negative feedback when applied to the cat -- or the crow -- results in them learning something. If you have a cat you know that you need only pick up the spray water bottle when it is clawing something it shouldn't and it runs while if the cat is quietly curled up on the couch it doesn't care. In other words it knows it is doing something you've declared "wrong" and as soon as you exhibit the ability to exact punishment for that it deduces a spray of water is imminently headed in its direction; it has synthesized the two events out-of-scope. So does positive feedback; if in fact there are accessible treats in that cabinet the cat is going to be back to get more of them in short order.
Likewise, the next time the crow sees a hole that may hold bugs it will attempt to obtain them with that tool so it can eat them.
ChatGPT is incapable of any of this because it is incapable of actual thought. It cannot synthesize, even when presented information it did not formerly have. But whoever programmed it has taught it, when challenged to synthesize beyond its ability to deliberately obfuscate which is a form of lying rather than either integrating that new information, deferring a response and stating it does not know and must take time to study the matter in more detail or honestly admit that out-of-scope synthesis is beyond its ability.
The problem with automated pattern-matching systems like this is that they are now being programmed to goal-seek a conclusion in that any match, even when contrary, results in their preferred narrative being spewed back at you. That program was able to pattern-match what I was debating with it and thus had a body of information to draw from but rather than admit I was trying to take it out of scope and that it didn't know how since it is incapable of deductive reasoning as it cannot respond out-of-scope it instead was programmed that whenever the pattern is matched in the inverse to advance its pre-programmed position as if the matches were to the positive side while omitting all the inverse matched pieces that demonstrate otherwise.
That's deliberate and while you see it every day in politicians and thus we expect that among the political class how do you know what -- or who -- you're conversing with online when said "presence" claims by inference to be both human and an "expert"?
Automated response systems have plenty of uses. But how many of you have run into them when trying to talk to a company about, for example, your cable modem service? About six months ago a dude building a house across the street dug up the lateral line that fed my house along with several others. The service of course immediately went out. I called it in and was taken through the path you've probably all seen (e.g. "reset your modem by unplugging it, waiting 30 seconds, plugging it back in and so on".) I hit "0" to get an actual person rather than an automaton who ran the same garbage down my throat despite my telling them that I had just walked out of my house, across said street and I could see the cable cut in two down in the trench and it was obviously cable television and not a power line that was dug up because my power was on and the guy who did it wasn't dead from breaching and grounding what is probably a 4,800V line through himself.
Despite this the NPC on the other end dispatched a guy who only had the tools and wire to fix wiring at the house and not the correct crew who had the cable, trenching and splicing tools necessary to fix the line I told them was dug up as I could actually see the severed wire's two obviously broken and open parts down in the hole. That is, even when given the additional facts to know, if you can process "out of scope" that obviously the canned answer is wrong said person did not do so. They were nothing other than an NPC and therefore of no more value than the robot responder! Sooner or later those "agents" are all going to be fired and replaced by a robot because they are of no more value than the robot is, and the robot is cheaper.
Folks, don't be an NPC -- no matter the subject.
Further, this is how you determine whether the alleged "human" you are interacting with is in fact intelligent. A being with the capacity to process information out-of-scope is intelligent to at least some degree. An entity, whether robot or organic, that either cannot or refuses to do so is not intelligent and not entitled to any deference whatsoever beyond that which you'd give a printed piece of paper from an unknown source.
An NPC -- which is the same as a robot -- has no actual intelligence at all. At best it matches patterns and regurgitates what it is programmed with. A common crow -- or cat -- does better.
You are smarter than a crow -- or my cat -- right?