ALT Large language models (LLMs) offer impressive performance in various zero-shot and few-shot tasks. However, their suc- cess in zero-shot and few-shot settings may be affected by task contamination, a potential limitation that has not been thoroughly examined. This paper investigates how zero-shot and few-shot performance of LLMs has changed chronolog- ically over time. Utilizing GPT-3 series models and several other recent open-sourced LLMs, and controlling for dataset difficulty, we find that on datasets released before the LLM training data creation date, LLMs perform surprisingly bet- ter than on datasets released after. This strongly indicates that, for many LLMs, there exists task contamination on zero- shot and few-shot evaluation for datasets released prior to the LLMs’ training data creation date. Additionally, we uti- lize training data inspection, task example extraction, and a membership inference attack, which reveal further evidence of task contamination.