Several benefits obtained using multiple robots in conversation have been reported in the human-robot interaction field. This paper first presents pre-trial results by which elderly people assigned a lower rating to a conversation with two robots than to one with a single robot. Observations of the trial suggest the hypothesis that an inappropriate spatial arrangement between robots and humans increases the workload in a conversation. Reducing the workload is important, especially when robots are used by elderly people. Therefore, we specifically examine the workload that is influenced by the spatial arrangement in group conversation. To verify the hypothesis, we use a NASA-TLX and a dual-task method to evaluate the workload and to conduct a comparative experiment in which the participant talks with two robots in two spatial arrangements. We also conduct a case study for elderly people in the same conversational conditions. From these experiments, we demonstrate that the spatial arrangement in which people cannot see both robots simultaneously increases their conversational workload and decreases their evaluation of the dialogue compared to a spatial arrangement by which people can see both robots simultaneously. We also show that the primary cause of the workload by positioning is not physical but mental.