I believe Talking About Large Language Models by Murray Shanahan should be required reading for anyone working with LLMs. You really must understand what you’re actually asking of a system vs. what it looks like you’re asking of a system.