Since the start of the reasoning models like OpenAI o1 I feel like we are making regress instead of progress. The models are ranked by some arbitrary benchmarks. Some benchmarks are very abstract like GSM8K - ranking how well models are doing grade school math (like yeah, most of the users of LLMs are not doing it for coding or marketing), but there are also more open benchmarks, built by community - not some university researchers - like for example LiveCodeBench using tasks i.e., from LeetCode. Here at least it is hard to overfit those models.
My experience sine the reasoning models? I hate it. Yes, there are tasks that are solved better by the reasoning LLMs - one-liner prompts, like “what is the difference between nextjs and laravel”. Even better, if you made a typo, because with the “more stupid” models this could easily derail your query. Is my laziness worth paying double the price? Maybe, but I started to ask myself - why I became so lazy?
Previously, I spent 10-15 minutes on the prompt I sent. Mainly, because I was planning to reuse it. I carefully picked the words and ordered steps. Why I do not do it anymore?
Mostly because with reasoning LLMs, it does not matter anymore. I no longer fully control reasoning directions, and models try to outsmart my prompts. Then when the query fails, it frustrates, in a way I feel dumb.
For commercial projects, where possible I still try to run pre-reasoning models - simply gives predictable results. However, most providers already deprecated non-reasoning models. Then I suggest to, minimize reasoning to reasoning\_effort = none.
And to put the last nail into gen ai coffin, last year, with $20/m subscription on LLMs I was able to do my work, and be happy. This year, $200/m subscriptions seems small, while results - well disputable.