Lets say we need multiple AI lines.
I think you're 150% right, Dr Dave.
Echoing your thoughts somewhat, what I think we need most of all is an innovative
pathfinding system that dynamically plots and chooses lines based on the location of other cars around it and the location of track grip. Need to go two or more wide through a corner or on a straight? Devise an algorithm that
takes the fast line and automagically plots a two-or-more wide path through the corner with sufficient room between cars (using car width in the calculation)and stays away from track boundaries -- and make sure it kicks in the moment the AI detects there is a car nearby, as if a spotter called it out to them (this is NASCAR Racing 2003's behaviour, interestingly). This would be useful for both side-by-side racing and overtaking manouevres of all kinds (including under multiclass and F1-style blue flags). Want the beautiful variability of lines that NR2003 AI run? Induce a little random variation in the radius of the line taken through the corner and braking points. And so on.
We have the statistical learning tools like neural networks to take some real-world training data from human sim racers from different hotlapping/racing scenarios and train the AI driving algorithm to drive the car and follow a specified line. This is a solved problem! And AC's AI shows this is possible even on player physics. The (large) challenge that remains for developers, then, is to thoughtfully design a versatile pathfinding algorithm that tells AI cars where they should go at any given time.
In the old days of Geoff Crammond's Grand Prix games and Papyrus Indycar and NASCAR, devs had to cheese the AI behaviour with quirky manual coding, in a kind of 'cop out' because computers couldn't possibly handle 43 cars of real-time physics. Makes for funny
examples like this. But this made the devs disciplined, and they manually coded in a lot of useful behaviour that still shines if you play the best of those sims nowadays. We have to return to that spirit of hard-coding behaviour IMO! Back to the future.
But we now have the advantage of multi-core CPUs that could run many AI cars' physics at once (unlike AC1's unfortunate choice to consign AI and player physics calculations to one thread) and advanced tools to train AI to drive with human-like inputs and results.