This paper proposes a fast method to robustly perform Visual Graph SLAM in underwater environments. Since Graph SLAM is not resilient to wrong loop detections, the key of our proposal is the Visual Loop Detector, which operates in two steps. First, a lightweight Siamese Neural Network performs a fast check to discard non loop closing image pairs. Second, a RANSAC based algorithm exhaustively analyzes the remaining image pairs and filters out those that do not close a loop. The accepted image pairs are then introduced as new graph constraints that will be used during the graph optimization. By executing RANSAC only on a previously filtered set of images, the gain in speed is considerable. The experimental results, which evaluate each component separately as well as the whole Visual Graph SLAM system, show the validity of our proposal both in terms of quality of the detected loops, error of the resulting trajectory and execution time.