Oil is one of the most important resources in the modern life. When an oil well is drilled, engineers extract the samples of core to analyze it and build the model of the geological formation. Now, the core samples and rock lithology segmentation is usually implemented by people by hand. Methods for the image segmentation and possible core samples segmentation approaches are reviewed. The novel dataset consisting of 69 images of segmented core samples created specifically for the task is presented in this paper. Also, two approaches for dataset creation were tried and described in this paper. The U-Net solution of the task with the first version of the dataset consisting of 4 classes and its results are described. Also the Mask R-CNN with ResNet-50 FPN model from the library Detectron2 with the second version of the dataset consisting of 11 classes of Argillite and Sandstone and its combination is described and results of experiments are provided.