I took a Udacity class by Norvig [1] and my abilities as a programmer clearly were improved afterward.

His code here demonstrates why. It is both shorter and much easier to understand than anything the LLMs generated. It is not always as efficient as the LLMs (who often skip the third loop by calculating the last factor), but it is definitely the code I would prefer to work with in most situations.

[1] https://www.udacity.com/course/design-of-computer-programs--...

  WITH all_numbers AS
  (
      SELECT generate_series as n
      FROM generate_series(1, 108) as n
  ),
  divisors AS
  (
      SELECT *
      FROM all_numbers
      WHERE 108 % n = 0
  ),
  permutations as
  (
      SELECT a.n as n1, b.n as n2, c.n as n3
      FROM divisors as a
        CROSS JOIN divisors as b
        CROSS JOIN divisors as c
  ) 
  SELECT *
  FROM permutations
  WHERE n1 * n2 * n3 =  108
      AND n1 < n2 And n2 < n3 
  ORDER BY  n1, n2, n3

I'm just here to point out that since Python 3.10, you don't need to import anything from typing anymore, you could use the built-in types for annotation:

  from math      import prod
  from itertools import combinations

  def find_products(k=3, N=108) -> set[tuple[int, ...]]:
    """A list of all ways in which `k` distinct positive integers have a product of `N`.""" 
    factors = {i for i in range(1, N + 1) if N % i == 0}
    return {ints for ints in combinations(factors, k) if prod(ints) == N}

  find_products()

It is more obvious when taken to extreme: With the current feedforward transformer architectures, there is a fixed amount of compute per token. Imagine asking a very hard question with a yes/no answer to an LLM. There are infinite number of cases where the compute available to the calculation of the next token is not enough to definitively solve that problem, even given "perfect" training.

You can increase the compute for allowing more tokens for it to use as a "scratch pad" so the total compute available will be num_tokens * ops_per_token but there still are infinite amount of problems you can ask that will not be computable within that constraint.

But, you can offload computation by asking for the description of the computation, instead of asking for the LLM to compute it. I'm no mathematician but I would not be surprised to learn that the above limit applies here as well in some sense (maybe there are solutions to problems that can't be represented in a reasonable number of symbols given our constraints - Kolmogorov Complexity and all that), but still for most practical (and beyond) purposes this is a huge improvement and should be enough for most things we care about. Just letting the system describe the computation steps to solve a problem and executing that computation separately offline (then feeding it back if necessary) is a necessary component if we want to do more useful things.

Tried these with some local models and these are the ones that generated the program one shot, a few of them also generated the results correctly one shot.

llama3.1-70b, llama3.1-405b, deepseekcoder2.5, gemma-27b, mistral-large, qwen2.5-72b. https://gist.github.com/segmond/8992a8ec5976ff6533d797caafe1...

I like how the solution sort of varies across most, tho mistral and qwen look really similar.