Go, PET, Let Hen - Curious adventures in (Commodore) BASIC tokenizing

Summary

Go, PET, Let Hen July 5, 2025 Curious adventures in (Commodore) BASIC tokenizing. No, this is not about a one-hit-wonder from the charts of 1977, rather, it’s about something that could have happened on a Commodore PET around this time. More generally, it is about a curious bug/feature in early BASICs and how this gave way to an additional keyword in Commodore BASIC for the PET. Both are related to BASIC syntax and how white space in BASIC text is handled by the interpreter. — Which sets up the topic of today’s installment, namely, the tokenizer routine in early MS 9-digit BASIC for the 6502 and how this evolved over time. As this promises some fun with #BASIC and #6502 code on the #PET and involves some #archeology, as well, we really ought to have a look at this… To begin with, let’s fire up a PET 2001 with BASIC 1.0, the first ROM version shipped with the PET 2001, also known as “Old ROM”, to start a little experiment: You can do so, as well, in emulation here. *** COMMODORE BASIC *** 7167 BYTES FREE READY. 10 IF LS = LE THEN GOTO 100 LIST 10 IF LS = LETHEN GOTO 100 READY. RUN ?SYNTAX ERROR IN 10 READY. █ So, what’s happening here?And why does a perfectly valid BASIC statement throw a sntax error? Let’s investigate… As you may probably know, all white space in BASIC is optional and we do not require any of this as a keyword delimiter or separator of any kind. The two following statements work perfectly the same: 100 FOR I = 0 TO 10 STEP 2 100FORI=0TO10STEP2 However, there’s actually a small difference: while the first one may be more readable, the second one executes a bit faster and requires less memory, for which it is preferred by the greybeards. Going back to our little experiment, something that may spring to the eye is a tiny, but crucial difference in the BASIC text as entered and the BASIC text as listed: 10 IF LS = LE THEN GOTO 100 LIST 10 IF LS = LETHEN GOTO 100 READY. Did you see it? — No, there is no paranormal entity. — Rather, it’s something missing...