In this work, we examine variations of the BERT model on the statute law retrieval task of the COLIEE competition. This includes approaches to leverage BERT's contextual word embeddings, finetuning the model, combining it with TF-IDF vectorization, adding external knowledge to the statutes and data augmentation. Our ensemble of Sentence-BERT with two different TF-IDF representations and document enrichment exhibits the best performance on this task regarding the F2 score. This is followed by a fine-tuned LEGAL-BERT with TF-IDF and data augmentation and our third approach with the BERTScore. As a result, we show that there are significant differences between the chosen BERT approaches and discuss several design decisions in the context of statute law retrieval.

Textual entailment classification is one of the hardest tasks for the Natural Language Processing community. In particular, working on entailment with legal statutes comes with an increased difficulty, for example in terms of different abstraction levels, terminology and required domain knowledge to solve this task. In course of the COLIEE competition, we develop three approaches to classify entailment. The first approach combines Sentence-BERT embeddings with a graph neural network, while the second approach uses the domain-specific model LEGAL-BERT, further trained on the competition’s retrieval task and fine-tuned for entailment classification. The third approach involves embedding syntactic parse trees with the KERMIT encoder and using them with a BERT model. In this work, we discuss the potential of the latter technique and why of all our submissions, the LEGAL-BERT runs may have outperformed the graph-based approach.

Under covert attention, our visual perception deteriorates dramatically as eccentricity increases. This reduction of peripheral visual acuity (PVA) is partially due to the coarse sampling of the retinal ganglion cells towards the periphery, but this property cannot be solely responsible. Other factors, such as character crowding, have been studied, yet the origin of the poor PVA is not entirely understood. This gap motivated us to investigate the PVA by varying the crowding conditions systematically. Under completely crowded conditions (i.e. resembling a full page of text), PVA was observed to be eight times worse than the PVA under uncrowded conditions. By partially crowding the periphery, we obtained PVA values between the fully crowded and uncrowded conditions. On the other hand, crowding the fovea center while leaving the periphery uncrowded improved PVA relative to the uncrowded case. These results support a model for a top-down "covert attention vector" that assists the resulting PVA in a manner analogous to saccadic eye movement for overt attention. We speculate that the attention vector instructs the dorsal pathway to transform the peripheral character to the foveal center. Then, the scale-invariant log-polar retinotopy of the ventral pathway can scale the centered visual input to match the prior memory of the specific character shape.