No, you're confusing GA with GP.

There's some minor issues with the blog post though that don't make it entirely correct Genetic Programming as written by Koza's. I'd have to dive into it again but I think it was either with the way mutations are done or how fitness is evaluated.

Anyway, it should be close enough.

> Also don’t confuse Genetic Algorithms (GA) with GP.

I just lump it all under "Evolutionary Computing"...

Real-time and energy-constrained problems would be the top use cases in my mind given the current state of affairs. Beyond this, problems that are non-differentiable or where we desire strong symbolic interpretation. I think these techniques also tend to extrapolate more robustly than DNNs. Technically, any problem where algorithmic shortcuts like the chain rule of calculus break down.

I've been working on GP architectures that use short linear program tapes as part of their genome. Executing these programs can be done very quickly. Assuming the program tape is <=10kb in length and you are constraining to ~a megacycle in the interpreter, you can execute these programs 100k~1m times per second on a chunky workstation. The problems you can solve in this time & space are non-trivial. Once you evolve the desired program, you can write the interpreter for it in any language on any platform within half an hour.

Training is definitely going to be expensive but it can fit in one powerful machine. You don't need to keep a terabyte of data around in VRAM to make reasonable rates of progress. Paging out parts of the population to disk or S3 buckets is feasible with these techniques. It subdivides very nicely into however many islands you can afford to run.

Inference is essentially free and could run on the processor in your toaster oven.

These approaches can make CUDA & friends look like a Rube Goldberg machine by comparison. Being able to explain how the entire thing works to a peer in a single afternoon is perhaps another significant feature of these designs.

At the end of the day, there is a pretty good reason these techniques aren't popular - There aren't any quadratic scaling laws that constrain the architecture of GP. Only the much scarier exponential ones. I contend that the search space of all linear programs is viable, but I still don't have any proof that this is the case after chasing the rabbit for about a year. My central argument is that there are many high-quality programs out there. We just need to find one of them. I think the arguments against this tend to unfairly cast the problem as searching for a single needle in a haystack of 10^10000.

Genetic Programming and the wider field of Evolutionary Computation and Genetic Algorithms are really good at some kinds of optimization.

If the problem fits into a tree form, then Genetic Programming programming is your best friend.

If the problem can be encoded onto a set of substrings of 0s and 1s, then Genetic Algorithms are your best friend.

Likewise if your problem can be encoded using floating point numbers, then Evolutionary Strategies is your best friend.

Now... I could ruffle some feathers by saying that Genetic Algorithms and Evolutionary Strategies aren't really all that different. Same algorithms could be used for both. But historicaly they they came roughly at similar times frome different places on earth, Illinois vs Germany.

Back to GP. The cool thing about GP is that when the solution is evolved you have THE solution to the problem. No futzing about with how or what the results mean.

A big problem in the past is that GP doesn't scale well like neural networks. It is not embarrassingly parallel. It has been very limited by the types of hardware/architectures available.

But GP is a great field of exploration!

Note that genetic programming is a specific subset of genetic algorithms that focuses on searching for "programs" (hence the name), typically encoded in the form of a tree structure similar to an AST, though other representations exist. In theory, you could use GP for almost any situation where you'd want to synthesize a mathematical expression or piece of code for which you have a criteria to optimize (i.e. "fitness"). In practice, since GPs are basically just semi-randomly searching a huge combinatorial space, they tend to work best in low-dimensional problems, ideally with a fitness function that is cheap to evaluate. They can work nicely for finding nonlinear symbolic formulas for regression, for example. But there's also some other cool results over the years - Hod Lipson has published several cool results in robotics with them.

Until a few years ago, the popular deep learning methods like CNNs weren't great at that kind of thing, but LLMs definitely changed that - it's safe to say that by drawing on huge amounts of data LLMs are much better at most practical programming tasks. They're not necessarily a complete replacement though, there's definitely space for hybrid approaches, eg https://arxiv.org/abs/2401.07102 or https://www.nature.com/articles/s41586-023-06924-6.

I suppose with Genetic Programming, given an appropriate set of descriptive symbols, it is relatively easy to understand the final result and intuit if there is any over-fitting involved. On the other hand, machine learning results are typically a black box, the weights involved typically do not easily lend themselves to understanding the nuances of the solution.

Genetic algorithms to fit car shapes: https://rednuht.org/genetic_cars_2/

Areas where the problem space is continuous but the solution model isn't differentiable.

An example would be a neural network over a discrete field.

Back at university I played around with boolean neural networks and this was one way of training them directly.

Definitely not an expert here, but the main difference is that in machine learning the goal is to find/optimize algorithm to fit given datasets, whereas in genetic programming the goal is to find/optimize dataset to fit given algorithm. There is a lot of overlap though.

EDIT: > what are the most suitable problems that Genetic Programming is to solve

Given above, genetic algorithms are really suitable when part of your problem domain is bounded execution time. You perform n iterations of genetic algorithm and even if the result does not fit within fitness function, you still get some result that is closer to fitness function than a wild guess.

Big combinatorial problems still use genetic algorithms. Specifically I know it's still used for logistics routing problems that are too big for solvers like Gurobi.

Deep learning on graphs is unfortunately still a little underwhelming.

I've used GP successfully in a very particular circuit path covering problem.

GA (not GP) is commonly used for hyperparameter choice in deep learning.

Well, the blog post is 14 years old.

It's an interesting showcase in how we (or at least I) assumed what the lifecycle of hyperlinks would be. (My assumption was they would perhaps 404 at some point, but not this.)

Scamers, spammers, online casinos, porn sites; all often take over expired domains to host their trash and collect stray traffic

Ants was really interesting. Humbling indeed.

Link to Ants AI Challenge: http://ants.aichallenge.org/

Winner's post mortem: https://web.archive.org/web/20151117155824/http://xathis.com...

The AI Challenges were so cool and I had hoped there would be more of them.

A "humbling experience" is definitely the right choice of words.

I did some minor work on a PoC for the one after Ants. It was multi-player (or: multi-bot) Asteroids but things fizzled out as most of the people organizing moved on to other things.

I thought it was a real baby.

[dead]