Rigorous Systems Research Group (RSRG) Seminar

The talk will present three tales told by Scheherazade to the Sultan in ArabIan Nights about AI and Science.

1. Sinbad and the Poisoned Datasets: There have been many cases where organizations with vested interests (e.g., tobacco companies) have looked to manipulate public opinion and policies towards their own interests. They have historically done so by bankrolling research designed to serve those interests. This was challenging and expensive. Given we are in this new AI age, what more can they do now, and can we mitigate that?
2. Ali Baba and the 40 Prompts: P-hacking involves researchers torturing data until they get desired -- but often spurious -- results. When using LLM as a judge or using LLMs for annotation, p-hacking is easily done by simply trying many prompts until a desired result is obtained. The conventional way of mitigating p-hacking is preregistration, where researchers must register their analysis plan before collecting any data. But that doesn't work here. So what can we do about it?
3. Aladdin and the Magic Latex: The Magic Latex (i.e., AI) has led to a rapid increase in the number of submissions. Some conferences and journals are putting fixed caps, e.g., nobody can submit more than a certain number of papers. But a large fixed cap leads to a ton of single-author submissions (e.g., in the TMLR journal we have seen many cases of 4-8 single-author submissions by the same person in a span of 1-2 weeks) and a small fixed cap means that advisors with multiple students cannot submit. So where do we draw the line? Or perhaps we draw a curve?