Despite the potential of robotic systems for automating the construction industry, the role of human operators remains essential for the success of these systems in complex and dynamic environments. However, current human-robot interfaces are often limited to low-level interactions that require constant micromanaging of robot movements. To address this limitation, this study proposes a deictic gesture-based interface that enables high-level task specification for construction robots. To evaluate the user experience and task performance of the proposed interface, we conducted a laboratory experiment with six human subjects who interacted with the robot to make openings in drywall panels. The results show that the proposed interface significantly reduced mental demand and effort levels among the participants compared to the conventional joystick interface. Moreover, task performance using the proposed interface was comparable in accuracy and efficiency to that achieved with the joystick interface. These findings highlight the potential of the proposed deictic gesture-based interface to facilitate intuitive human-robot interaction and precise operation of construction robots, particularly in situations where as-planned building models are not readily available.