Feature Creep 2026, 06 - A Mental Math Trainer - Tokenizer and Debugging the Tests

The easiest way to turn what I have into a tokenizer is to make every symbol its own "entity" by turning the exception of the string into an array of stuff. That's really as easy as declaring it differently, calling append() instead of +, and reassembling the entire thing before sending it to the front-end. I never really need it as a string, unless in the tests, which would then also get a function_assembler(function_array) helper:

def function_assembler(self, function_array):
  retstr = ''
  for i in function_array:
    retstr += i
  return retstr

Then, if I need to insert a token at a given place, that should be pretty easy and I'm not worrying about whether I'm splitting up some numbers at some point. I can also create a sanity check for the brackets that's less cursed.

def brackets_check(self, f_arr):
  cas_count = [0, 0, 0, 0, 0, 0]
  bracket_types = ['(', ')', '[', ']', '{', '}']
  if i in f_arr:
    if i in bracket_types:
      cas_count[bracket_types.getIndex(i)] += 1

  if (cas_count[0] != cas_count[1])
        || (cas_count[2] != cas_count[3])
        || (cas_count[4] != cas_count[5]):
    return False
  return True

The question of how to automatically correct bracket mismatch remains to be tackled, but hopefully I've written my generators in a way where I won't need the check in earnest.

Tokenizing the equation generator will make generating numbers a wee bit more complicated. I want to have a fair number of natural numbers kicking around, so I need to make a case to the script that sometimes generating those as a random number is a good idea. As such, I'll make three different functions, for number-tokens.

def gen_natural_num(self, min, max):
  return random.randrange(min, max)

def gen_fractional_num(self, min, max, rounder):
  nonnull = random.randrange(self, min, max)
  if nonnull == 0:
    nonnull = 1
  return round(random.randrange(self, min, max)/nonnull, rounder)

def gen_float_num(self, min, max):
  return random.uniform(self, min, max)

Two of these are fairly trivial, the other one makes a lot of assumptions, but it'll be fine for my purposes. This application doesn't actually benefit from non-pseudo-randomness either way. This will do away from the idea of inserting an operator wherever the script decides it could go, but rather it'll have to track whether the last addition to the equation was an operator or what I'm going to start calling a "constituent" of the equation. The constituents will include numbers, functions sometimes, a termination marker that will close all the open brackets.

I had trouble setting up the project structure, and then some venv configurations were really, really wrong, including that the pytest binary refused to point to the one in the venv, so I only got to debugging the tests this week. It was a genuine annoyance, and I really, really don't like how rigid all this stuff has gotten since python3. It's kind of indicative of the messiness going on underneath the whole packaging infrastructure. Relative imports, specifically are just a straight-up horrifying mess now. I'll have to think about the directory structure again soon, because of me needing to move all the subdirectories into an additional src directory to stop compiler from throwing a fit.