I took a Udacity class by Norvig [1] and my abilities as a programmer clearly were improved afterward.

His code here demonstrates why. It is both shorter and much easier to understand than anything the LLMs generated. It is not always as efficient as the LLMs (who often skip the third loop by calculating the last factor), but it is definitely the code I would prefer to work with in most situations.

[1] https://www.udacity.com/course/design-of-computer-programs--...

  WITH all_numbers AS
  (
      SELECT generate_series as n
      FROM generate_series(1, 108) as n
  ),
  divisors AS
  (
      SELECT *
      FROM all_numbers
      WHERE 108 % n = 0
  ),
  permutations as
  (
      SELECT a.n as n1, b.n as n2, c.n as n3
      FROM divisors as a
        CROSS JOIN divisors as b
        CROSS JOIN divisors as c
  ) 
  SELECT *
  FROM permutations
  WHERE n1 * n2 * n3 =  108
      AND n1 < n2 And n2 < n3 
  ORDER BY  n1, n2, n3

I'm just here to point out that since Python 3.10, you don't need to import anything from typing anymore, you could use the built-in types for annotation:

  from math      import prod
  from itertools import combinations

  def find_products(k=3, N=108) -> set[tuple[int, ...]]:
    """A list of all ways in which `k` distinct positive integers have a product of `N`.""" 
    factors = {i for i in range(1, N + 1) if N % i == 0}
    return {ints for ints in combinations(factors, k) if prod(ints) == N}

  find_products()

It is more obvious when taken to extreme: With the current feedforward transformer architectures, there is a fixed amount of compute per token. Imagine asking a very hard question with a yes/no answer to an LLM. There are infinite number of cases where the compute available to the calculation of the next token is not enough to definitively solve that problem, even given "perfect" training.

You can increase the compute for allowing more tokens for it to use as a "scratch pad" so the total compute available will be num_tokens * ops_per_token but there still are infinite amount of problems you can ask that will not be computable within that constraint.

But, you can offload computation by asking for the description of the computation, instead of asking for the LLM to compute it. I'm no mathematician but I would not be surprised to learn that the above limit applies here as well in some sense (maybe there are solutions to problems that can't be represented in a reasonable number of symbols given our constraints - Kolmogorov Complexity and all that), but still for most practical (and beyond) purposes this is a huge improvement and should be enough for most things we care about. Just letting the system describe the computation steps to solve a problem and executing that computation separately offline (then feeding it back if necessary) is a necessary component if we want to do more useful things.

Tried these with some local models and these are the ones that generated the program one shot, a few of them also generated the results correctly one shot.

llama3.1-70b, llama3.1-405b, deepseekcoder2.5, gemma-27b, mistral-large, qwen2.5-72b. https://gist.github.com/segmond/8992a8ec5976ff6533d797caafe1...

I like how the solution sort of varies across most, tho mistral and qwen look really similar.

It's worth noting that math and programming do not appear to be considered "languages" by much of the academic and/or neuroscientific literature; see [0] on the front page right now and my comments regarding the same [1].

[0] https://news.ycombinator.com/item?id=41868884

[1] https://news.ycombinator.com/item?id=41892701

  Saussure approaches the essence of language from two sides. For the one, he borrows ideas from Steinthal and Durkheim, concluding that language is a 'social fact'. For the other, he creates a theory of language as a system in and for itself which arises from the association of concepts and words or expressions. Thus, language is a dual system of interactive sub-systems: a conceptual system and a system of linguistic forms. Neither of these can exist without the other because, in Saussure's notion, there are no (proper) expressions without meaning, but also no (organised) meaning without words or expressions. 

What about o1? I think the world is sleeping on o1. Recently I misread a leetcode/neetcode problem (so I was curious that my version of the problem with an extra constraint could be solved in a different way). And 4o hallucinated incorrectly and double downed when asked follow up questions - but o1 solved it the first time what seemed like easily. It really is a major step forward.

    import itertools

    def factors(n):
        result = set()
        for i in range(1, int(n**0.5) + 1):
            if n % i == 0:
                result.add(i)
                result.add(n // i)
        return sorted(result)

    def main():
        n = 108
        factors_list = factors(n)
        triplets = []
        for a_idx in range(len(factors_list) - 2):
            a = factors_list[a_idx]
            for b_idx in range(a_idx + 1, len(factors_list) - 1):
                b = factors_list[b_idx]
                for c_idx in range(b_idx + 1, len(factors_list)):
                    c = factors_list[c_idx]
                    if a * b * c == n and a < b < c:
                        triplets.append((a, b, c))
        print("All ways in which three distinct positive integers have a product of 108:")
        for triplet in triplets:
            print(triplet)

    if __name__ == "__main__":
        main()

    -▶ python a.py
    All ways in which three distinct positive integers have a product of 108:
    (1, 2, 54)
    (1, 3, 36)
    (1, 4, 27)
    (1, 6, 18)
    (1, 9, 12)
    (2, 3, 18)
    (2, 6, 9)
    (3, 4, 9)

    % Define a predicate to find all positive divisors of N
    divisors(N, Divisors) :-
        findall(D, (between(1, N, D), N mod D =:= 0), Divisors).

    % Define a predicate to generate all combinations of three distinct elements from a list
    combination3(List, X, Y, Z) :-
        select(X, List, Rest1),
        select(Y, Rest1, Rest2),
        select(Z, Rest2, _),
        X < Y, Y < Z.

    % Define a predicate to find triplets (X, Y, Z) such that X*Y*Z = N
    find_triplets(N, X, Y, Z) :-
        divisors(N, Divisors),
        combination3(Divisors, X, Y, Z),
        X * Y * Z =:= N.

    % Define a predicate to list all the triplets
    list_triplets(N) :-
        findall((X, Y, Z), find_triplets(N, X, Y, Z), Triplets),
        writeln(Triplets).

    ?- [a]. list_triplets(108).
    [(1,2,54),(1,3,36),(1,4,27),(1,6,18),(1,9,12),(2,3,18),(2,6,9),(3,4,9)]

Related HN discussion - https://news.ycombinator.com/item?id=41831735 using Prolog as an intermediate target language for LLM output improves their 'reasoning' abilities.

A formal notation for reasoning could possibly solve some reasoning issues for LLMs. Perhaps something like Lojban or symbolic logic. We don't have a lot of data for it, but it might be possible to synthetically generate it.

On a dark note, I wonder if increasing AI reasoning capability could have dangerous results. Currently, LLMs are relatively empathetic, and seem to factor the complex human experience into it's responses. Would making LLMs more logical, cold, and calculating result in them stepping on things which humans care about?

Gut feel: doing this in two steps (1. write an algorithm for and 2. write code for) or even chain-of-thought prompting might yield better results.

> The language that a problem-solver uses matters!

Because the "intelligence" is borrowed from language (lower entropy)

It's not the language it's the context. LLMs could give very different outputs depending on the supplied context. In this case, words "python" and "program" put it in a far better context to solve the problem.

I have question, we still do not know what behind "neuron network" of machine. What if we gave them some syntax of a language (or syntax of a area) then ask them extract a general rule for that language ? Can them can do that ?

I thought this was going to be an essay on the impact of English, Math and Programming on humanity.

I would place English (or all spoken/written languages in general) first, Math (as discovered) second, and programming languages last.

>> Only 2 of the 9 LLMs solved the "list all ways" prompt, but 7 out of 9 solved the "write a program" prompt. The language that a problem-solver uses matters! Sometimes a natural language such as English is a good choice, sometimes you need the language of mathematical equations, or chemical equations, or musical notation, and sometimes a programming language is best. Written language is an amazing invention that has enabled human culture to build over the centuries (and also enabled LLMs to work). But human ingenuity has divised other notations that are more specialized but very effective in limited domains.

If I understand correctly, Peter Norvig's argument is about the relative expressivity and precision of Python and natural language with respect to a particular kind of problem. He's saying that Python is a more appropriate language to express factorisation problems, and their solutions, than natural language.

Respectfully -very respectfully- I disagree. The much simpler explanation is that there are many more examples, in the training set of most LLMs, of factorisation problems and their solutions in Python (and other programming languages), than in natural language. Examples in Python etc. are also likely to share more common structure, even down to function and variable names [1], so there are more statistical regularities for a language model to overfit-to, during training.

We know LLMs do this. We even know how they do it, to an extent. We've known since the time of BERT. For example:

Probing Neural Network Comprehension of Natural Language Arguments

https://aclanthology.org/P19-1459/

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

https://aclanthology.org/P19-1334/

Given these and other prior results Peter Norvig's single experiment is not enough, and not strong enough, evidence to support his alternative hypothesis. Ideally, we would be able to test an LLM by asking it to solve a factorisation problem in a language in which we can ensure there are very few examples of a solution, but that is unfortunately very hard to do.

______________

[1] Notice for instance how Llama 3.1 immediately identifies the problem as "find_factors", even though there's no such instruction in the two prompts. That's because it's seen that kind of code in the context of that kind of question during training. The other LLMs seem to take terms from the prompts instead.

so, in conclusion, the training data containing 'math' that LLMs have access to is predominantly written as software code, and not as mathematical notation

Within hours to weeks to months, many LLMs will be better at solving the problem as originally given in TFA ... because of the existence of TFA.

Not immediately clear what this means or if it is good or bad or something else.

When an author writes things like:

"But some of them forgot that 1 could be a factor of 108"

I struggle to take them seriously. The anthropomorphization of AI into something that can "know" or "forget" is ridiculous and shows a serious lack of caution and thinking when working with them.

Likewise it leads people into wasting time on "prompt engineering" exercises that produce overfit and worthless solutions to trivial problems because it makes them feel like they're revealing "hidden knowledge."

Genuinely disappointing use of human time and of commercial electricity.