As I am helping Interactive Surfaces and Spaces conference as an Associate Chair this year, I figured it would be good to summarize a bit the general process and how I go about it. Comments welcome, they are open!
If you land here before having read Ken Hinckley’s “So you’re a Program Commitee Member Now”. You should definitely read it first. This post is more of a personal account.
In the following I’ll likely use AC and PC member interchangeably. AC stands for Associate Chair and PC for Program committee, as ISS shifts to a journal model it’s even called an Editorial Board this year, hence the initials AE (associate editor I guess) you might also see below. While I’m spelling out acronyms, PCS is the submission and reviewing system we use.
I was invited by the chairs at the beginning of April. Nic and Fanny gave a super clear overview of the conference reviewing timeline and the workload. It’s the second time I’m doing it for ISS, a conference I really enjoy attending and submitting to.
I guess the first decision happens at this stage. Whether or not you accept the invitation. The review process happens during French holidays which is not great. But given my teaching load during the fall, I would probably say no to most invitations during the period from September to February.
Here is the reviewing timeline we followed:
- Jul 8: Authors’ deadline
- Before July 8: PC members upload a collection of sample papers on PCS.
- July 9-10: PC members declare conflicts and bid on submissions (see instructions below)
- July 10-13: Chairs assign papers
- Jul 14: PC members start assigning external reviewers (1AE: 1 reviewer, 2AE: 1 reviewer)
- Jul 23: All papers should have 2 externals
- Aug 6: External reviews due, 2AC personal review due
- Aug 7: 1AE / 2AE start meta reviews, discussion starts; 3AE requests start
- Aug 14: Discussion ends, 3AE reviews due
- Aug 17: 1AC Meta-review due, 1AE/2AE/3AE tentative recommendation
- Aug 21: Notifications sent to authors
- Aug 21-Sep 14: Authors revise papers
- Sep 14-28: Second round of reviews
- Sep 28: Individual reviews due
- Sep 28: Discussion starts
- Oct 3: Meta-reviews and 1AE/2AE final recommendation due
- Oct 5: Notifications sent to authors
- Oct 9: Camera ready due
Workload for ISS
This year ISS is moving closer to a journal like submission process similar to Ubicomp and CSCW. Papers submitted can be accepted, asked for minor revisions, major revisions, or rejected. Papers with majors revisions have to be submitted to the next review cycle (right now, it’s quite close to a full reject).
I reviewed 3 papers and meta-reviewed 3. For each paper, I had to find one reviewer, the other AC on the paper finding the other, leading to 3 reviews + 1 meta-review.
Bidding on papers and managing conflicts
Before the paper deadline, we were supposed to upload some of our own papers. I don’t know if it was for the chairs or for PCS to compute some sort of research profiles/keywords (I’d be somewhat skeptical of this).
After the paper deadline we had two days to bid on papers and declare conflicts. In PCS, we could see a list of people with affiliations and declare whether or not we were in conflict with them.
The ACM has a policy on what constitutes a conflict: https://www.acm.org/special-interest-groups/volunteer-resources/acm-conflict-of-interest-policy. I have a slightly more expansive definition of conflict that includes personal ties. In the end it depends of our appreciation, the chairs sent us as well a list of common conflict cases:
- employment at the same institution or company
- candidate for employment at the same institution or company
- received an honorarium or stipend from the institution or company within the last year
- co-author on book or paper in the last 48 months
- co-principal investigator on grant or research project
- actively working on project together
- family relationship
- close personal relationship
- graduate advisee/advisor relationship
- deep personal animosity
Once you have declared conflicts you cannot get the papers from those people. You are ready to bid on papers. At this stage you can see a list of all the submissions in PCS, and can select one of the following option:
- want: you really want to review this submission, and are qualified to do so
- willing: you are qualified to review this submission, and are willing to do so, though not your top pick
- blank (no option selected): will be considered as a soft willing.
- reluctant: you are not qualified to review this submission.
- conflict: automatically checked for you, after you declare all of your conflicts.
The chairs invited us to select ~10-15 papers as “want” and at least 10 papers as “willing”, so that they could pick from at least 20 papers for each of us. Unfortunately I didn’t keep track of the papers I selected as want or willing to compare them to my assigned papers.
The summer made it maybe a bit harder to get reviewers, or a least to get rapid accept or decline answers.
I typically have an idea of who to invite from the title and abstract. Especially with the bidding phase filtering out papers, it is quite straightforward to think of a few people to invite for reviewing. I still scan the paper to get make sure I know the type of paper it is (system, study, design, theory oriented), and check the references to see if the authors are noticeably citing someone (or not citing people I thought would be relevant). This helps me decide in which order I’ll invite potential reviewers.
One paper turned out to be tricky, five people declining to review or not answering. I asked an old colleague now based in Asia if he had reviewer recommendations on the topic coming from this part of the world where I know less people, and then checked who would be most relevant for the paper. This worked quite well !
After the invitations, I turned to my reviews on the 3 papers I was AC2. These are “normal” reviews. The only difference I see is that my recommendation for acceptance/revision/reject would necessarily trigger a discussion if it contradicted the recommendation of the primary AC.
Discussions and meta-reviews
After all the papers have their reviews in (I was personally late for some, sorry co-ACs). The discussion and meta-reviewing process starts. One possibility here is to ask for an extra person to review the paper. Given the tight schedule this was probably quite rare (I don’t know how many papers got extra reviews at this stage).
Here starts the synthesis of the reviews. As an AC, I sorted out and weighted the reviewers remarks. In the six papers I was involved in, the reviewers came to the similar conclusions, with different angles and attention to different aspects. But there wasn’t clear conflicts about the quality of a paper (or all the reviewers going in a direction opposite to mine). This made it particularly straightforward to write the meta-reviews, and made our discussions quite simple. For one paper I had a short video-call with the AC to discuss it, and emailed the chairs to discuss the final recommendation.
Finally, I also had to remind some reviewers to ensure that their reviews did not have vague claims about the literature, the supposed knowledge of the authors on topic, or the ability of the authors.
The paper chairs made the final decisions based on our recommendations, and sent them out within a few days. They exchanged with the ACs in case of problem, but not with me (only got an email later to wrapping things up).
The final step will be to check the papers accepted with minor revisions to check if the authors made them.